A younger son of the 2nd Marquess of Bristol and Lady Katherine Isabella Manners, he was educated at Balliol College, Oxford, where he graduated with a Master of Arts. Hervey was Justice of Peace for Suffolk and Honorary Fellow in Hertford College, Oxford. He was Member of Parliament (MP) for Bury St Edmunds from 1874 to 1880 and from 1885 to 1892. Between 1892 and 1907, Hervey was Civil Service Commissioner and between 1907 and 1909 First Civil Service Commissioner.