Bergen was elected as a Democrat to the Thirty-ninth Congress, holding office from March 4, 1865 to March 3, 1867. He was not a candidate for renomination in 1866 and resumed agricultural pursuits and surveying near New Utrecht, and also engaged in literary and historical work. He served as ensign, captain, adjutant, lieutenant colonel, and colonel of the Two Hundred and Forty-first Regiment, New York State Militia (known as Kings County Troop). In 1881 Bergen died in Brooklyn; interment was in Green-Wood Cemetery.
Bergen's second cousin, John Teunis Bergen, was also a U.S. Representative from New York.