Born in Marylebone, Cavendish was the third son of William Cavendish, 7th Duke of Devonshire, by his wife, Lady Blanche Howard (a daughter of the 6th Earl of Carlisle and a niece of the 6th Duke of Devonshire). His father and his two surviving brothers were all Members of Parliament (MPs): his eldest brother Spencer, MP for North Lancashire 1857–91 and later 8th Duke of Devonshire, led the Liberal Party and was asked three times to be Prime Minister by Queen Victoria; the middle brother, Frederick was MP for the West Riding and Chief Secretary for Ireland and was assassinated in 1882.
Cavendish was commissioned into the Rifle Brigade. On retirement from the regular army he joined the 2nd Derbyshire Militia (later the Sherwood Foresters), but transferred to the King's Own Royal Lancaster Regiment in 1888.
Cavendish served as MP for West Derbyshire. Like his eldest brother, he became a Liberal Unionist in reaction to the Liberal Party's policy on Irish Home Rule. Upon his death in Marylebone aged 53, his son, Victor, was elected unopposed to replace him, later becoming the 9th Duke of Devonshire.