During his thirty-five year career, Brumfield won 4,573 races in 33,222 rides. He retired from racing in 1989. Brumfield was the "track all-time leading rider in terms of races won (925)" at Churchill Downs, where he won 16 riding titles. His record was later broken by Pat Day, who won more than 2,000 races at Churchill Downs in his career. Brumfield rode Kauai King to victory in the 1966 Kentucky Derby.
He was inducted into the National Museum of Racing and Hall of Fame in 1996.