Asked by Terre Haute politician and lawyer John Palmer Usher, future Secretary of the Interior under President Abraham Lincoln, to testify in a slander case, Cannon became fascinated with the law. Eventually, he asked Usher if he could study law under him and moved to Terre Haute. At age 19 he traveled to Cincinnati, Ohio to attend a semester of law school at the University of Cincinnati law school.
He was admitted to the bar in 1858 and commenced practice in Terre Haute, Indiana, in 1858, but was disappointed when Usher refused to offer him a place in his office. That year he relocated to Tuscola, Illinois. His choice of a new hometown was somewhat involuntary, taking place whilst he was travelling from Shelbyville, Illinois, to Chicago to find more clients for his law firm. During the trip, he ran out of money. He boarded a Chicago-bound train in Mattoon, Illinois; after the train had started, he was asked for his ticket. Because Cannon did not have a ticket, he was removed from the train in Tuscola. There, he became State's attorney for the twenty-seventh judicial district of Illinois, holding the position from March 1861 to December 1868. In 1876 moved to Danville, Illinois, where he resided the rest of his life. He and his wife Mary P. Reed, whom he married in 1862, had two daughters.
Cannon, a member of the Republican Party, was elected as to the United States House of Representatives from Illinois to the Forty-second and to the eight succeeding Congresses (March 4, 1873–March 4, 1891), and was the chairman, Committee on Expenditures in the Post Office Department (Forty-seventh Congress), Committee on Appropriations (Fifty-first Congress).
He attempted to gain the Speakership four times before succeeding. His antic speaking style, diminutive stature and pugnacious manner were his trademarks. The newspapers frequently lampooned him as a colorful rube.
Joseph was chairman to the Committee on Appropriations (Fifty-fourth through Fifty-seventh Congresses), Committee on Rules (Fifty-eighth through Sixty-first Congresses), and Speaker of the House of Representatives (Fifty-eighth through Sixty-first Congresses). He received fifty-eight votes for the presidential nomination at the Republican National Convention at Chicago in 1908.
In 1910 an Insurgent revolt flared in the House as both Democrats and dissatisfied Republicans stripped the Speaker of some of his powers, such as heading the House Rules Committee and ability to appoint members of other House committees.
Cannon was defeated in 1912 but returned in 1914 and was re-elected through 1922. He was a critic of President Woodrow Wilson and US entry into World War I. He was also an outspoken critic of Wilson's League of Nations.
Cannon retired in 1922; he was featured on the cover of the first issue of Time magazine on the last day of his term in office.
Born a Quaker, he became a Methodist after leaving Congress. However, he may have been effectively a Methodist long before this. After marrying Mary Reed in a Methodist service 1862, a Quaker encouraged him to express regret for this, to which Cannon replied, "If you mean that I am to get up in meeting and say that I am sorry I married Mary, I won't do it. I'm damned if I'm sorry and I'm damned if I will say I am."
Joseph Cannon died in his residence in Danville, Vermilion County, Illinois. He had a weakened heart and also suffered from the general effects of old age. Cannon expired at noon on November 12, 1926 while in a deep sleep. He was buried in Spring Hill Cemetery.