His wife, Henrietta Maria (21 December 1807 – 16 February 1895), a daughter of Viscount Dillon, was a remarkable woman. Born in Halifax, Nova Scotia she was a great-great-great-great-granddaughter of King Charles II by his mistress Barbara Villiers. Before her marriage in 1826 she had lived in Florence, and had attended the receptions of the Countess of Albany, the widow of Charles Edward, the Young Pretender; and in London she had great influence in social and political circles. When he was patronage secretary her husband was described by Lord Palmerston as joint-whip with Mrs Stanley. Later in life Lady Stanley of Alderley helped to found the Women's Liberal Unionist Association, and she was a strenuous worker for the higher education of women, helping to establish Girton College, Cambridge, the Girls' Public Day School Trust, and the Medical College for Women. She died on 16 February 1895.
They had ten children