Born in Hoboken, New Jersey, DeCarlo represented Genovese family business interests in the New Jersey underworld; he was an associate of Abner "Longey" Zwillman and Gerardo Catena. DeCarlo was an excellent cook; his specialty was fried pasta with anchovies. He reportedly married a woman named Frances Ryan and had two children. His arrest record included income tax evasion, breaking and entering, counterfeiting, robbery, and liquor law violations. DeCarlo owned the La Martinique Tavern in the suburban Newark, New Jersey community of Mountainside, New Jersey
Based in New Jersey, DeCarlo rose to the position of caporegime, or captain, of a crew. He achieved this promotion by gaining control over the loansharking, or "juice" racket, illegal gambling, and murder for hire. DeCarlo also systematically eliminated his rival mobsters during the late 1950's.
Between 1961 and 1963, federal agents began wire tapping conversations between DeCarlo and mob associates. These taps revealed corruption among law enforcement, prominent businessmen and state officials, including New Jersey Congressmen Peter Rodino, Newark Mayor Hugh Addonizio and influential Hudson County politician John J. Kenny. Both Addonizio and Kenny lost their careers when the wire transcripts were published. The transcripts also mentioned Sinatra. DeCarlo was also overheard discussing methods for fulfilling murder contracts; he described a so-called "humane" hit in which he shot a victim through the heart. According to DeCarlo, it was supposedly a painless way to die.
In 1970, DeCarlo was convicted of extortion related to the arsenic poisoning of Louis Saperstein and was sentenced to 12 years imprisonment. However, after serving a year and a half, President Richard Nixon pardoned DeCarlo due to his poor health. DeCarlo himself claimed to be dying of cancer. The pardon request was approved by U.S. Attorney General Richard Kleindienst, who submitted it to White House Counsel John Dean, who delivered it directly to Nixon. It was rumored that Nixon was influenced to grant the pardon by Vice President Spiro Agnew, a personal friend of DeCarlo. Sinatra). Although the pardon was investigated by Special Watergate Prosecutor Archibald Cox, no evidence was found of corruption.