"Yo mama" jokes evolved out of a social game commonly called the Dozens, in which participants come up with creative insults to demean each other jokingly in front of a crowd. The tradition grew out of verbal games played by slaves in pre-Civil War America to relieve stress and frustration.
Originally, these insult games were similar in theme to a comic roast, in which participants made exaggerated insults about each other's most prominent characteristics. The shift to attacking relatives might have come about to rile up players that were good at brushing off personal insults, increasing the emotional stakes of the game. Losing one's temper during a game of the Dozens is often viewed as a failure, so the more insults piled upon a player's beloved mother, the harder it would be for him to keep his wits.
Comic insults and similar contests are common in many different cultures, with notable examples appearing in the time of Shakespeare in Europe. However, the practice of insulting relatives, and particularly mothers, is most commonly seen in the African-American tradition.
The origin of the term "the Dozens" is unknown. Popular theories include the use of 12-part rhymes in the game, or the belief that players must come up with 12 distinct insults to participate.