James was the third son of General Sir Ralph Abercromby of Tullibody, who fell at the battle of Alexandria, March 28, 1801, by a daughter of John Menzies of Fernton, Perthshire. She was created Baroness Abercromby.
Prior to his elevation to the peerage he was appointed, in 1830, Lord Chief Baron of the Court of Exchequer in Scotland, Master of the Mint (1834) in the administration of Lord Grey, and was elected Speaker of the British House of Commons on February 19, 1835 where he sat until moving to The House of Lords.
In 1841 Lord Dunfermline was elected as Dean of Faculty in the University of Glasgow.
He married, June 14, 1802, Mary Anne, daughter of Egerton Leigh, Esq., of West Hall, in High Legh. Their son Ralph, KCB (b.1803), was sometime Secretary of Legation at Berlin, and minister plenipotentiary and envoy extraordinary to Sardinia (1840-1851), and thereafter The Hague.