Just after the start of Lt. Gen. Ulysses S. Grant's Overland Campaign in 1864, Ledlie transferred to the Army of the Potomac, commanding a brigade in Maj. Gen. Ambrose Burnside's IX Corps. He assumed command of the 1st Division on June 9. It was in this command that his brief military career was ruined.
During the Siege of Petersburg, former coal miners in Burnside's corps devised an ingenious plan to lift the stalemate by digging a 511-foot tunnel under the Confederate lines. On July 30, 1864, they detonated the explosives, creating a crater some 135 feet in diameter that remains visible to this day. Some 250 to 350 Confederate soldiers were instantly killed in the blast. The Union plan was to exploit the explosion by sending well-rehearsed African-American troops of Edward Ferrero's division into the gap and driving for critical objectives deep in the Confederate rear area. The plan was modified at the last minute, however, due to political concerns about the effect the black troops would have on the Confederate soldiers and the public in general. Burnside, despondent at the change in plans, resorted to a lottery to select a replacement division. Ledlie drew the short straw and disaster resulted. He did not brief his troops beforehand and they entered the crater out of curiosity instead of moving safely around its rim, as Ferrero's division had been trained to do. Unable to exit the steep sides of the crater, they were slaughtered by Confederates firing down on them. Over 5,300 Union troops were casualties in the ill-fated battle that achieved none of its objectives. But most damning for Ledlie's reputation was the fact that he did not lead, or even accompany, his men into battle. He and Ferrero were observed behind the lines in a bunker, drinking liquor.
Ledlie was criticized by a court of inquiry into his conduct that September and, in December, he was effectively dismissed from the service by Maj. Gen. George G. Meade, on orders from Gen. Grant. He formally resigned his commission on January 23, 1865.
Ledlie died in New Brighton, Staten Island, and is buried in Forest Hill Cemetery, Utica, New York.