Key was born in Greene County, Tennessee, the son of Reverend John and Margaret (Armitage) Key. In 1826 the family removed to Monroe County where Key was reared, graduating from Hiwassee College in 1850. He selected the legal profession as his vocation through life, and the same year of his graduation was admitted to the bar. For two years he practiced law at Madisonville, then a short time at Kingston, and in February, 1853, moved permanently to Chattanooga. In 1857 Key married Elizabeth Lenoir, with whom he had nine children.
When the Civil War broke out, Key enlisted in the Forty-third Confederate Tennessee Regiment of Infantry, served until the close of the war, and was mustered out as a lieutenant colonel. He then resumed the practice of law until 1868.
Key was a member of the Tennessee state constitutional convention of 1870, which composed the basic instrument of government of the state still in effect, and in August of the same year was elected chancellor of the Chattanooga (3rd) division. He maintained his chancellorship during an unsuccessful campaign for the U.S. House in 1872 but he resigned in 1875 to accept the appointment by governor of Tennessee James D. Porter to a vacant Senate seat caused by the death of Andrew Johnson. Defeated in the next election in the Tennessee General Assembly, in 1877 he was appointed Postmaster General by President Hayes, and filled the office until August 25, 1880. His appointment to the position of Postmaster General was a part of the Compromise of 1877 to ensure Democratic power in the Republican cabinet. Later he accepted appointment as federal judge for the U.S. District Court for the Eastern District of Tennessee and U.S. District Court for the Middle District of Tennessee. He died in Chattanooga and is buried there.