He was born in Jamaica, where his father, Robert Scarlett, had property. In the summer of 1785 he was sent to England to complete his education at Hawkshead Grammar School and afterwards at Trinity College, Cambridge, taking his B.A. degree in 1789. Having entered the Inner Temple he was called to the bar in 1791, and joined the northern circuit and the Lancashire sessions.
Though he had no professional connections, he gradually obtained a large practice, ultimately confining himself to the Court of King's Bench and the northern circuit. He took silk in 1816, and from this time till the close of 1834 he was the most successful lawyer at the bar; he was particularly effective before a jury, and his income reached £18,500, a large sum for that period.
He first entered parliament in 1819 as Whig member for Peterborough, representing that constituency with a short break (1822–1823) till 1830, when he was elected for the borough of Malton. He became Attorney-General, and was knighted when Canning formed his ministry in 1827; and though he resigned when the Duke of Wellington came into power in 1828, he resumed office in 1829 and went out with the Duke in 1830.
His opposition to the Reform Bill caused him to leave the Whigs and join the Tories, and he was elected, first for Cockermouth in 1831 and then in 1832 for Norwich, for which he sat until the dissolution of parliament in 1835. He was appointed Lord Chief Baron of the Exchequer in 1834, and presided in that court for more than nine years. While attending the Norfolk circuit on April 2, he was suddenly seized with apoplexy, and died in his lodgings at Bury.
He had been raised to the peerage as Baron Abinger in 1835, taking his title from the Surrey estate he had bought in 1813. The qualities which brought him success at the bar were not equalled on the bench; he had a reputation for unfairness, and complaints were made about his domineering attitude towards juries.
Lord Abinger was twice married (the second time only six months before his death), and by his first wife (d. 1829) had three sons and two daughters, the title passing to his eldest son Robert (1794–1861). His second son, General Sir James Yorke Scarlett (1799–1871), leader of the heavy cavalry charge at Balaklava, is dealt with in a separate article; and his elder daughter, Mary, married John Campbell, 1st Baron Campbell of St Andrews, and was herself created Baroness Stratheden (Lady Stratheden and Campbell) (d. 1860). Sir Philip Anglin Scarlett (d. 1831), Lord Abinger's younger brother, was chief justice of Jamaica.