He married firstly on 4 August 1632 (when both were children, he was c11), Mary daughter of Thomas Smith of Soley in Chilton Foliat, a village north west of Hungerford, on the border of Berkshire and Wiltshire. By her he had one daughter, Frances.
He married secondly in 1654, Elizabeth Alington (baptised 1635, died 1691), daughter of William Alington, 1st Baron Alington of Killard (14 March 1610/11, d. c. October 1648). They had three children.
Both of Charles's sons, Francis and Charles, ultimately succeeded to the dukedom of Somerset, his own father having been a younger brother of William Seymour, 2nd Duke of Somerset (aka marquis of Hertford). His daughter, Honora Seymour, married Sir William Gerard, 3rd Baronet.
Charles Seymour died aged 45 and was succeeded in the barony by his elder son, Francis.