He was born in 1788 in Charleston, South Carolina, the grandson of American Revolutionary patriot Christopher Gadsden. He received his bachelor's degree from Yale University in 1806. After a career as a US Army officer, James was appointed a commissioner in 1823 to assist the government in moving the Seminoles to reservations. He served as president of the South Carolina Railroad Company from 1840 to 1850 and promoted the construction of a transcontinental railroad by the southern route. In 1853, he was appointed U.S. minister to Mexico to negotiate the Gadsden Treaty which led to the Gadsden Purchase by the United States from Mexico of about 30,000 square miles in the southern section of what is now Arizona and southwestern New Mexico.