Like many other African-American artists from the era, Simon began singing in his father's baptist church. He pursued his vocal abilities full-time once the family moved to Richmond, California (near Oakland) in the late 1950s. There Simon joined the Golden West Gospel Singers and became influenced by the popular Sam Cooke and Arthur Prysock. With this, the group decided to turn secular and recorded "Little Island Girl" as the Golden Tones in 1959.
Hush Label Record owners Gary & Carla Thompson urged Simon to record on his own and in 1964 Simon scored considerable success on the Vee-Jay label with "My Adorable One". Simon scored big in 1965 on the Chicago-based label with "Let's Do It Over", which landed a #13 spot on Billboard's R&B charts. However, the Vee-Jay label folded soon after the latter song's release and Simon found himself traveling across the country singing anywhere he could.
Simon caught the eye of Nashville R&B disc jockey John Richbourg during this time, and Richbourg not only became Simon's manager/producer but also brought the singer to Monument Records' subsidiary label Sound Stage 7 in 1966. That year Simon released "Teenager's Prayer", which peaked at #11 on Billboard's R&B charts. Within the next two years, Simon released a string of hits: "(You Keep Me) Hanging On", "The Chokin' Kind" (Pop #13), "Farther On Down The Road", and "Yours Love".
Under the encouragement of Richbourg, Simon moved to the Polydor-distributed Spring Record Label in 1970, which paired Simon with hitmakers Kenny Gamble and Leon Huff. The team scored a #3 R&B hit in 1971 with "Drowning In The Sea of Love" and a #1 R&B hit in the summer of 1972 with "Power Of Love". Both songs landed #11 spots on the Pop Charts. Simon continued to release R&B hit after hit with "Pool Of Bad Luck", "Trouble In My Home", "Step By Step", "I Need You, You Need Me", "Music In My Bones", "Carry Me", and 1975's "Get Down, Get Down (Get On The Floor & Let the Good Times Roll)", which gave Simon his third #1 R&B hit, and also a #8 Pop hit. Joe Simon's success escalated with his writing/producing the theme for the film Cleopatra Jones in 1973.
In the late 1970s/early 1980s, Joe Simon decided to take his tenor/bass-baritone voice away from the secular music world and devote it and his life to religion. Simon began evangelist preaching in Flossmoor, Illinois. In 1982/1983, he produced the album Lay My Burden Down for former Davis Sisters second lead Jackie Verdell. Simon released a gospel LP titled This Story Must Be Told in the mid-late 1990s.
Simon has had a number of his songs sampled by other artists, including OutKast, who sampled "Before the Night is Over" in their hit "So Fresh, So Clean" and Lil' Kim, who sampled Simon's "It Be's That Way Sometimes" in "Magic Stick", featuring 50 Cent.