He married Elizabeth Sutherland, 19th Countess of Sutherland, daughter of William Sutherland, 18th Earl of Sutherland and the former Mary Maxwell, on 4 September 1785. They had four surviving children:
He was invested as a Privy Counsellor in 1790, a Knight of the Garter in 1806 and was created Duke of Sutherland on 28 January 1833. He is perhaps most well-known for his role in carrying out the Highland Clearances, where thousands of tenants, including the elderly and infirm, were forced out of their homes, which were often burned down, in order to make way for sheep.
The clearances, described by some commentators as a form of ethnic cleansing, were undertaken between 1811 and 1820. At first they involved relocations from Assynt to coastal villages on the assumption that farmers could take up fishing. Later when the consequences of these actions became clear, the evictions were met with opposition, which was ruthlessly repressed. Resentment mounted when one of his factors was acquitted of murder and then took over one of the massive sheep farms the evictions created. Condemnation was widespread and the Highlanders' grievances were heard in the British House of Commons. However, little was done in practice to prevent the emptying of the glens.
In 1837 a large monument, known locally as the Mannie, was erected on Ben Bhraggie near Golspie to commemorate the Duke's life. The existence of this statue has been the subject of some controversy—in 1994, Sandy Lindsay, a former Scottish National Party councillor from Inverness proposed its demolition. He later altered his plan, asking permission from the local council to relocate the statue and replace it with plaques telling the story of the Clearances. Lindsay proposed moving the statue to the grounds of Dunrobin Castle, after the J. Paul Getty Museum in Los Angeles declined his offer to take it. As of July, 2008, however, the statue still stands.