Small grew up in suburban New Rochelle, New York. He graduated from New Rochelle High School followed by Brown University in 1963 where he earned a Bachelor's Degree in Spanish literature. He held executive positions in Citicorp and Citibank before becoming the President and COO of the FNMA, a position he held between 1991 to 2000. He is a director of the Chubb Corporation and serves on the boards of Marriott International, Inc., New York City's Spanish Repertory Theatre, the National Gallery of Art, the John F. Kennedy Center for the Performing Arts and the Woodrow Wilson International Center for Scholars.
Small's plans were met with considerable resistance. In 2001, Small withdrew his proposal to close the Smithsonian National Zoological Park's Conservation and Research Center in Front Royal, Virginia after strong opposition from scientists.
In 2006, the Smithsonian agreed to a deal with Showtime to create Smithsonian Networks, a joint venture to create television programming with the Smithsonian's resources. Controversy over the deal emerged when it was disclosed that the network had the right of first refusal for commercial documentaries that rely heavily on Smithsonian collections or staff.
In 2007, Acting Smithsonian Inspector General A. Sprightley Ryan reviewed Small's expenses and reported that $90,000 of expenses between 2000 and 2005 were unauthorized, prompting the watchdog group Citizens for Responsibility and Ethics in Washington to call on the Attorney General Alberto R. Gonzales to investigate whether the spending violated federal law. The U.S. Senate froze a $17 million appropriations increase for the Smithsonian, citing Small's compensation as excessive. On March 26, Small resigned from his position.
A subsequent independent report commissioned by the Smithsonian was highly critical of Small's "imperialistic and insular" management style, including documented resistance to sharing information with the Regents and the audit committee. It also disputed claims earlier made on Small's behalf (see above) on the extent of his personal fund-raising achievements. The Smithsonian in response acknowledged the need to institute reforms in its governance and oversight arrangements.