Born in New York City and raised in Newark, New Jersey, Grachan began playing the cello at age nine, and switched to the trombone at eleven. In high school he attended the Laurinburg Institute in North Carolina, the private school where Dizzy Gillespie had studied. While still in school he began sitting in with touring jazz musicians on their way through town, such as Art Blakey and Jackie McLean, with whom he formed a lasting friendship.
After high school he toured with Ray Charles (1959–1962), Art Farmer's and Benny Golson's Jazztet (1962), and Sonny Rollins. He took part in two classic Jackie McLean albums in the early 1960s, One Step Beyond and Destination Out, to which he also contributed the bulk of compositions and which led to two influential albums of his own for Blue Note Records, Evolution (1963) with Jackie McLean and Lee Morgan, and Some Other Stuff (1964) with Herbie Hancock and Wayne Shorter.
After leaving Blue Note, Grachan joined Archie Shepp's ensemble and recorded with other avant-garde players such as Marion Brown, Beaver Harris and Roswell Rudd (the other big name in free jazz trombone). During a stay in Paris in the summer of 1969, he recorded two albums as a leader for the famous BYG Actuel label, New Africa and Aco Dei de Madrugada, as well as appearing as a sideman on numerous other releases of the label. In 1974, the Jazz Composers' Orchestra of America commissioned him to write Echoes of Prayer (1974), a jazz symphony featuring a full orchestra plus vocalists and jazz soloists. His sixth album as a leader, Shadows (1977) was released only in Japan. Unfortunately, he was subsequently plagued by health problems and copyright disputes and recorded only rarely. Through the 1980s he recorded with Cassandra Wilson (1985), played occasionally with the Paris Reunion Band and Frank Lowe, appeared on John Patton's Soul Connection (1983), but mostly concentrated on teaching. In 2004 he re-emerged with a new album (Exploration) on Capri Records featuring Grachan's compositions arranged by Mark Masters for an octet including Tim Hagans and Gary Bartz.