Born in Fredericksburg, Virginia, Mercer graduated from Princeton College in 1797, where he later took a postgraduate course and received his degree in 1800. He studied law and was admitted to the bar in 1802, commencing practice in Aldie, Virginia, a village that he founded in 1810. He was a member of the Virginia House of Delegates from 1810 to 1817 and was appointed a lieutenant colonel of a Virginia regiment in the War of 1812. Mercer was later promoted to major in command at Norfolk, Virginia, was inspector general in 1814, aide-de-camp to Governor James Barbour and brigadier general in command of the 2nd Virginia Brigade.
He was projector and first president of the Chesapeake & Ohio Canal Co. from 1828 to 1833 and was a delegate to the Virginia Constitutional Convention in 1829. Mercer was elected a Federalist, Crawford Republican, Adams Republican, Anti-Jacksonian and Whig to the United States House of Representatives in 1816, serving from 1817 to 1839. There, he served as Chairman of the Committee on Roads and Canals from 1831 to 1839. He was one of the originators of the plan for establishing the Free State of Liberia, and in that context one of the founders of the American Colonization Society in 1816, was vice president of the Virginia Colonization Society in 1836, and vice president of the National Society of Agriculture in 1842.