Born in Shreveport, Louisiana, he began his professional baseball career in with the Monroe Monarchs, a minor Negro League team. In , he signed with the Kansas City Monarchs, for which he played continuously until seeing action in World War II in 1944-45. During his pre-war baseball years, he established himself as having the most raw power in Negro League history, and possibly in the history of baseball. He hit home runs more often than the better known Josh Gibson, causing Gibson to give Brown his nickname. He also hit for a batting average of .374 in and regularly hitting over .350. Brown was one of the fastest players in baseball in the late 1930s and 1940s,as well as a solid outfielder.
He played briefly in the major leagues, in , having signed with the floundering St. Louis Browns, but he struggled because of the racism inherent in his new surroundings and his frustration with the lack of talent of his new teammates. After batting .179 in 21 games, he left the majors, but not before becoming the first black player to hit a home run in the American League. That winter, he went to Puerto Rico and had one of his greatest seasons ever, batting .432 with 27 home runs and 86 RBI in just 60 games, winning the Triple Crown and earning the nickname Ese Hombre or "That Man".
He also won the Puerto Rican Winter League Triple Crown in the 1949-1950 season.
He also hit for the cycle once in his career.
He returned to the Monarchs for the 1948 season and stayed with the team until his retirement from top-level baseball in . Subsequently, he was a successful minor league player in the Texas League from through .
Although his career home run total is not known, there is no doubt that he was among the Negro League career leaders in homers, despite a relatively brief career.
After he retired from baseball, he lived in Houston, Texas, and died there at age 81.
Known statistics: .355 career batting average.