Paine was born in Chardon, Ohio, and was the first cousin of future general Eleazar A. Paine. After attending the common schools, he graduated from Western Reserve College in 1845 and moved to Mississippi for a year to teach school. He returned to Cleveland to study law. He passed his bar exam in 1848 and established a practice. He moved to Milwaukee, Wisconsin, in 1857 and continued his legal career.
With the outbreak of the Civil War, Paine entered the Union army as the colonel of the Fourth Wisconsin Volunteer Regiment. He was promoted to brigadier general in March 1863 and led the Third Division of the Army of the Gulf in an assault on Priest Gap during the Battle of Port Hudson, where he suffered a wound that necessitated the amputation of his leg. After his recovery, Paine commanded troops in the defenses of Washington, D.C. during Jubal A. Early's raid in 1864. He was brevetted major general of volunteers on March 13, 1865. He resigned from the army on May 3 and returned to Wisconsin.
Paine, a Republican, was elected to Congress, serving from December 1865 until March 1871. He was a delegate to the Philadelphia loyalists' convention of 1866. In 1869, he championed the passage of a bill that provided for taking meteorological observations in the interior of the continent. He served as chairman of the Committee on Militia (Fortieth Congress), and the Committee on Elections (Forty-first Congress). After the expiration of his third term in Congress, he retired from politics and chose not to accept renomination.
Paine then practiced law in Washington, D. C. for several years before accepting President Ulysses S. Grant's appointment as the United States Commissioner of Patents in 1879, serving in that post for two years. In 1888, he authored Paine on Contested Elections. He died in 1905 in Washington D.C. and was buried in Arlington National Cemetery.