Born in Paducah, Kentucky in McCracken County, Kentucky, Carroll continued to reside in that county, where he worked as a lawyer. He served as a member of the Kentucky House of Representatives, serving five terms beginning in 1961 representing Paducah. Carroll served as Speaker of the House from 1968 through 1970. In 1971 when former governor Bert T. Combs sought a second term as Governor of Kentucky, he selected Carroll as his informal running mate. Candidates for governor and lieutenant governor of Kentucky ran separate from each other until the passage of a 1992 constitutional amendment; still, Combs, facing a formidable challenge from his one-time protege Wendell H. Ford, sought to create a balanced ticket to his advantage in the Democratic primary. Combs lost to Ford, but Carroll won the separate primary for Lieutenant Governor of Kentucky and then won the general election. Carroll took office as Lieutenant Governor of Kentucky in late 1971. Carroll later succeeded to office as Governor of Kentucky when Governor Ford resigned in 1974 upon his election to the United States Senate. Carroll became the first Kentucky governor ever from the state's far-western Purchase region. He was subsequently elected to a full term after landslide victories in the 1975 Democratic primary and general elections.
Carroll's administration increased funding for primary and secondary schools, created the Department of Energy, and advocated the use of coal to offset the energy crisis. He also served as Chairman of the National Governors Association in 1978. After his service as governor Carroll resumed his law practice in Frankfort, Kentucky for many years. He unsuccessfully sought another term as governor in 1987, finishing fifth in the Democratic primary behind Lieutenant Governor Steve Beshear, former governor John Y. Brown, Jr., Grady Stumbo, and the winner, Wallace G. Wilkinson. Carroll again returned to his Frankfort law practice. In 2004, Carroll was elected to the Kentucky State Senate from the 7th district which includes the state capitol of Frankfort, Kentucky.