McHugh came from a theatrical family. His parents ran a stock theatre company and as a young child he performed on stage. His brother Matt and sister Kitty performed an act with him by the time he was ten years old, but the family quit the stage around 1930. His brother Jimmy McHugh became a composer who wrote "On The Sunny Side of the Street" and "I'm in the Mood for Love" among many other songs, and another brother, Ed, became an agent in New York.
Frank debuted on Broadway in The Fall Guy in 1925. Warner Bros. hired him as a contract player in . McHugh played everything from lead actor to sidekick and would often provide comedy relief. He appeared in over 150 films and television productions and worked with almost every star at Warner Bros. By the 1950s, his film career had begun to decline, as evinced by his smaller role in Career (). From to , he played the television role of Willis Walter on The Bing Crosby Show.