Anderson-Pelham was the son of Francis Anderson and his wife Eleanor (née Carter). Born Charles Anderson he assumed the additional surname of Pelham, which was the maiden name of his paternal grandmother. He was elected to the House of Commons for Beverley in 1768, a seat he held until 1774, and then represented Lincolnshire until 1794. The latter year he was raised to the peerage as Baron Yarborough, of Yarborough in the County of Lincoln.
Lord Yarborough married Sophie, daughter of George Aufrere, in 1770. She died in 1786. Yarborough survived her by over forty years and died in September 1823, aged 74. He was succeeded in the barony by his son Charles, who was created Earl of Yarborough in 1837.