Lichfield was the eldest son of Thomas Anson, 1st Earl of Lichfield, and his wife Louisa Catherine (née Philips). He was returned to Parliament for Lichfield in 1847, a seat he held until 1854, when he succeeded his father in the earldom and took his seat in the House of Lords. He also served as Lord-Lieutenant of Staffordshire from 1863 to 1871.
Lord Lichfield married Lady Harriett Georgiana Louisa, daughter of James Hamilton, 1st Duke of Abercorn, in 1855. They had nine sons and four daughters. He died in January 1892, aged 66, and was succeeded by his eldest son Thomas. Lady Lichfield died in 1913.