Sanders moved to New York City in 1961 after playing with rhythm and blues bands. He received his nickname "Pharoah" from Sun Ra, with whom Sanders performed. He came to prominence playing with John Coltrane's band starting in 1965, as Coltrane began adopting the avant-garde jazz of Albert Ayler, Ra and Cecil Taylor. Sanders first performed on Coltrane's Ascension (recorded in June 1965), then famously on their dual-tenor recording Meditations (recorded in November 1965). After this Sanders joined Coltrane's final quintet, usually performing very lengthy, dissonant solos. Coltrane's later style was strongly influenced by Sanders.
In the 1970s, Sanders pursued his own recordings and continued to work with the likes of Alice Coltrane on her Journey In Satchidananda album. Most of Sanders' best-selling work was made in the late 60's and early 70s for Impulse Records, including the 30-minute wave-on-wave of free jazz "The Creator has a Master Plan" from the album Karma. This featured Sanders key musical partner, pianist Lonnie Liston Smith, who worked with Sanders from 1969-1971. Other members of his groups in this period include bassist Cecil McBee and vocalist Leon Thomas, on albums such as Jewels of Thought, Izipho Zam, Deaf Dumb Blind and Thembi.
In 1994 he traveled to Morocco to record with master Gnawa musician Maleem Mahmoud Ghania, resulting in the Bill Laswell-produced The Trance Of Seven Colors. Sanders continued to work with Laswell, Jah Wobble and others on the albums Message From Home (1996) and Save Our Children (1998). In 1999, he complained in an interview that despite his pedigree, that he had trouble finding work:
"I don't work that much myself. I would love to work, but nobody calls me. I have to just rely on and pray that I work somewhere.
AAJ: Why do you feel you are not getting work?
PS: I think that it maybe the agencies keep me from working. I have asked many, many times, but I don't know. What is the point? I don't have nothing personal against anybody. I feel like, maybe, it's me. Maybe it's the music, or maybe it's the way I express myself. I know that it may be hard for somebody to listen to me play the way I play, but I don't know. I'm just going to keep on playing and hope that I can do better, play better, and keep learning. I can only try to live healthy and try to live life.