Although born in Poulan, Georgia, Heflin was the nephew of prominent Alabama politician James Thomas Heflin and greatnephew of Alabama congressman Robert Stell Heflin. Following graduation from high school, Heflin attended Birmingham-Southern College (graduated 1942) and the University of Alabama Law School (graduated 1948). During this time he also served as a U.S. Marine officer in World War II and was awarded the Silver Star for combat, seeing action on Bougainville and Guam.
He became a law professor, and then became the Chief Justice of the Alabama Supreme Court from 1971 to 1977.
In 1978, Heflin was elected as a Democrat to the United States Senate to succeed John Sparkman. He remained in the Senate, where he rose to become Chairman of the Select Committee on Ethics, until January 3, 1997. While on the Ethics Committee, he led the prosecution against fellow Senator Howard Cannon (D-NV) for violations of Senate rules.
His stances on cultural issues most often reflected the region he was from. He strongly opposed legal abortion and all gun control laws. Heflin supported prayer in public schools and opposed extending federal laws against discrimination to lesbians and gays. He voted in favor of the Gulf War and against limiting spending on defense. With Fritz Hollings from South Carolina, he was one of only two Democrats in the Senate to vote against the Family and Medical Leave Act. He occasionally voted with Republicans on taxes. On other economic issues he was more in sync with the populist wing of his party. He voted against the North American Free Trade Agreement (NAFTA), General Agreement on Tariffs and Trade (GATT), and attempts to weaken enforcement of consumer protection measures. He strongly supported affirmative action laws. He memorably voted against the nomination of Clarence Thomas to the Supreme Court, complaining of his lack of experience and interest.
During his tenure, Heflin was considered to have bipartisan support if he were nominated for a vacancy on the United States Supreme Court by President Ronald Reagan. Nevertheless, Heflin did not wish to serve on the highest court in the United States.