Nabrit III was born in Texas to James Nabrit, Jr., a prominent civil rights attorney, law professor and later President of Howard University. James Nabrit III graduated from Bates College in Lewiston, Maine in 1952 and Yale Law School in 1955. Nabrit began his career with the law firm of Reeves, Robinson & Duncan, served two years in the Army and then spent 30 years (1959–89) as an attorney with the NAACP Legal Defense and Educational Fund, Inc. He argued many important civil rights cases before the U.S. Supreme Court and various U.S. Court of Appeals, including Swann v. Charlotte Mecklenburg Bd. of Ed. in 1972 <402 U.S 1> and Shuttlesworth v. Birmingham in 1969.