Smith grew up in Washington, D.C. and attended Phillips Academy, where he was the only African-American in the Class of 1955. He received an A.B. degree from Yale University in 1959 and an LL.B. from Yale Law School in 1962. In 1961, William Sloane Coffin invited second-year law student Smith to go to Montgomery, Alabama as a Freedom Rider. He and ten other Freedom Riders were arrested in the Montgomery bus station and convicted of breach of the peace; their convictions were later reversed by the United States Supreme Court.
Smith was a judge of the Civil Court of New York City from 1975-86 and an associate justice of the Supreme Court of New York from 1987-92. Governor Mario Cuomo appointed him to a 14-year term on the New York Court of Appeals in September 1992. Although politicians such as David Dinkins and Charles Rangel urged outgoing Governor George Pataki to reappoint Smith, he nominated Eugene F. Pigott, Jr. to the seat.
In December 2005, Smith was awarded the William Nelson Cromwell Award by the New York County Lawyers Association.
Smith's twin sister, Inez Smith Reid, is a judge of the District of Columbia Court of Appeals.