In 1848, he was a Whig member of the Massachusetts Senate. From 1849 to 1853, Devens was United States Marshal for Massachusetts, in which capacity he was called upon in 1851 to remand the fugitive slave, Thomas Sims, to slavery. This he felt constrained to do, much against his personal desire; subsequently, he attempted in vain to purchase Sims' freedom, and many years later appointed him to a position in the United States Department of Justice in Washington, D.C..
Devens was promoted to brigadier general of volunteers in April 1862 and assigned command of the 1st Brigade/1st Division/IV Corps. He was again wounded at the Battle of Fair Oaks in May. His brigade was not heavily involved in the Maryland Campaign. Shortly afterwards, it was reassigned to the VI Corps. Devens commanded the 2nd Brigade/3rd Division/VI Corps during the Battle of Fredericksburg.
Assigned command of the first division in the XI Corps, Devens was again wounded, this time at the Battle of Chancellorsville in May 1863. His inattention of the exposed right flank of the division helped Stonewall Jackson launch his flank attack on the corps. Devens's failure to react to early warnings of Jackson's flanking movement may have been due to his heavy drinking in an attempt to dull the pain of a previous injury.
Devens later distinguished himself at Battle of Cold Harbor, while commanding the 3rd Division/XVIII Corps in Ulysses S. Grant's Overland Campaign. During final stages of the Siege of Petersburg, he commanded the 3rd Division of the XXIV Corps.
Devens' troops were the first to occupy Richmond after its fall in April 1865.
Devens was also a key figure in the investigation into the unlawful execution of Confederate veteran Calvin Crozier by soldiers of the 33rd Regiment, U.S. Colored Troops, at Newberry, SC in September 1865 following an altercation. Over Devens' strong objections the officer who took responsibility for the lynching was exonerated and returned to duty.
He was a judge of the Massachusetts superior court, from 1867 to 1873, and was an associate justice of the Massachusetts Supreme Judicial Court from 1873 to 1877, and again from 1881 to 1891. From 1877 to 1881, he was Attorney General of the United States in the Cabinet of President Rutherford B. Hayes.
Fort Devens in central Massachusetts, which opened in 1917, was named after him, as was its successor, the Census-designated place Devens, Massachusetts. A statue of him stands outside the Devens Court House.