Carswell was born in Irwinton, Georgia, graduated from Duke University in 1941, and briefly attended the University of Georgia School of Law before joining the United States Navy. He served as a lieutenant in the Naval Reserve and was discharged in 1945 at the end of World War II. Carswell completed his legal education at the Walter F. George School of Law of Mercer University in 1948. Griffin B. Bell, 72nd Attorney General of the United States, was a classmate at Mercer.
Carswell served as a private attorney in Tallahassee, Florida, from 1948 to 1953 and as United States Attorney for the Northern District of Florida from 1953 to 1958. In 1958, he was nominated and confirmed as a judge of the United States District Court for the Northern District of Florida. In 1969, he was nominated and confirmed as a judge of the United States Court of Appeals for the Fifth Circuit.
On January 19, 1970, after Clement Haynsworth was rejected by the U.S. Senate for an appointment to the United States Supreme Court, President Richard Nixon nominated Carswell to be an Associate Justice of the Supreme Court to replace liberal Justice Abe Fortas. Carswell was praised by Southern Senators including Richard B. Russell, Jr., but was criticized by others for the high reversal rate (58%) of his decisions as a District Court Judge. Civil-rights advocates also questioned his civil rights record; in 1948, Carswell had voiced support for racial segregation while running for a seat in the Georgia state legislature (in his hometown, Irwinton, Georgia; Carswell did not win the election and moved to Florida where he started his career as a private attorney).
In defense against charges that Carswell was "mediocre", U.S. Senator Roman Hruska (Republican, Nebraska) stated, "Even if he is mediocre, there are a lot of mediocre judges and people and lawyers. They are entitled to a little representation, aren't they, and a little chance? That remark is believed to have backfired and damaged Carswell's cause.
On April 8, 1970, the United States Senate refused to confirm Carswell's nomination to serve on the Supreme Court. The vote was 51 to 45. 17 Democrats and 28 Republicans voted for Carswell. 38 Democrats and 13 Republicans voted against him. President Richard Nixon accused Democrats of having an anti-Southern bias as a result. He then nominated Harry Blackmun to fill the Fortas vacancy. Blackmun was later easily confirmed.
Carswell resigned from the Court of Appeals on April 20, 1970, and entered the political arena. In 1970, he ran unsuccessfully for a seat in the United States Senate from Florida, losing in the Republican Party primary by a large margin to U.S. Representative William Cramer, who went on to lose the general election to Democrat Lawton Chiles .
In 1976, Carswell was arrested and convicted of battery for advances he made to an undercover police officer in a Florida men's room; some claim him as the first homosexual or bisexual nominated to the Supreme Court. Carswell subsequently withdrew from public life. He returned to the private practice of law and then lived in retirement until his death.