He married Caroline Margaret Clayton (1821-1904), daughter of Sir William Clayton MP, at Gretna Green, Scotland in 1840, and they had six children:

• Lady Gertrude Georgiana Douglas (d. 1893), married Thomas Stock
• John Sholto Douglas, 9th Marquis of Queensberry (1844–1900), the man behind the Marquess of Queensberry rules that formed the basis of modern boxing, and, much later, the downfall of author and playwright Oscar Wilde
• Lord Francis William Bouverie Douglas (1847–1865), beaten by A. W. Moore and party by a day to first ascent of the Ober Gabelhorn, killed a week later in the first successful ascent of the Matterhorn
• Rev. Lord Archibald Edward Douglas (1850–1938)
• Lady Florence Caroline Douglas (1855–1905) (twin), war correspondent, travel writer, and feminist
• Lord James Edward Sholto Douglas (1855–1891) (twin), married Martha Lucy Hennessy in 1888, committed suicide

He died while hunting in 1858, officially from the explosion of his gun; the event was widely believed to be a suicide.

|}