The band was formed in Harlem in 1930, with reedman Bingie Madison the first of its many leaders. It was temporarily known as the Coconut Grove Orchestra, changing to Mills Blue Rhythm Band when Irving Mills became its manager in 1931. At various times the same group was known as the "Blue Rhythm Band", "Blue Ribbon Band", "Blue Rhythm Boys", "King Carter's Royal Orchestra", "Mills Music Masters", "Harlem Hot Shots and uncredited playing behind Louis Armstrong.
The Mills Blue Rhythm Band were based at The Cotton Club in New York. They were the backup band for when the Duke Ellington and Cab Calloway Orchestras were touring. Drummer Willie Lynch led the band until 1931, when Baron Lee took over as leader. Edgar Hayes, Eddie Mallory and Dave Nelson all had temporary stints as band leader until Lucky Millinder took over the role in 1934.
The band recorded 150 sides for a variety labels including Brunswick, Columbia, Victor, Oriole and Melotone. Despite success with a few hit records (including "Truckin'" and "Ride, Red, Ride") and a lineup of talented soloists, the group never became one of the more popular bands of the day. This is often attributed to the lack of a single identifable leader, and Irving Mills' preference to have the band perform an understudy role. Many of the Mills Blue Rhythm Band's recordings are now considered jazz classics by collectors.
The group disbanded in 1938. Millinder joined Bill Doggett's band before reforming it into his own orchestra in 1940