He was born in Derby, Vermont in 1827, the son of Moses French Colby, and came to Stanstead, Quebec with his family in 1832. He studied at Dartmouth College in New Hampshire. He studied law, was called to the bar in 1855 and entered practice at Stanstead. Colby was elected as a Liberal-Conservative MP in the Canadian House of Commons in 1867 representing Stanstead and remained in parliament until his defeat in 1891. He served as President of the Privy Council under Sir John A. Macdonald from 1889 to 1891 and was previously Deputy Speaker and Chairman of Committees of the Whole of the House of Commons. Colby supported the introduction of tariffs to reciprocate against those imposed by the United States. He was a trustee of Stantead College and a director for several railway companies. Colby served as vice-president of the Quebec Temperance and Prohibitory League.
In 1859, Colby built Carrollcroft, his residence at Standstead, which now serves as the site of the Colby-Curtis Museum.