From 1931 to 1935, he was an assistant United States Attorney in the Southern District of New York. From 1935 to 1947, he worked under Thomas E. Dewey and District Attorney Frank S. Hogan. He assisted in the prosecution of such notorious gangsters as Waxey Gordon and Louis (Lepke) Buchalter. In 1942 he was appointed Chief of the Homicide Bureau in the New York County District Attorney's Office. He resigned in 1948 to go into private practice. He was appointed the 17th Fire Commissioner of the City of New York by Mayor Vincent R. Impellitteri on July 18, 1951 and served in that position until the end of the Impellitteri Administration on December 31, 1953.
Appointed Chairman of the State Commission of Investigation by Governor Nelson A. Rockefeller in 1960, Grumet led an inquiry into charges of corruption in New York City that helped usher in the mayoral election year of 1961, in which Robert F. Wagner was re-elected.
Grumet, a Republican, left the panel in 1968 to become a State Supreme Court Justice. He returned in 1976 to head an inquiry into a charge by a former special prosecutor, Maurice H. Nadjari, that Governor Hugh L. Carey had tried to dismiss him because he had been investigating high-level Democrats. The charge was dismissed for lack of evidence.