Before the Dissolution of the Monasteries in 1536 the land on which Holker stands belonged to Cartmel Priory. When the land came up for sale it was acquired by the Preston family, who were established local landowners. Since then it has passed from the Prestons to the Lowthers and then the Cavendishes by inheritance, but it has never been sold.
The Prestons are believed to have made Holker their main residence in about 1610. The estate was briefly sequestered by the Parliamentarians during the Civil War, but was restored upon the payment of fine. The last owner from the Preston family died in 1697 and left the house to his daughter Catherine, who was married to Sir William Lowther. Sir William's son Sir Thomas laid out formal gardens and added a north wing to the house.
The third and last Lowther to own the property, another Sir William, died childless before the age of thirty and left the estate to his cousin Lord George Augustus Cavendish, a younger son of the 3rd Duke of Devonshire, who served for a time as Comptroller of the Royal Household, and in his old age became Father of the House of Commons. Lord George Augustus commissioned John Carr to make alterations to the east and north wings of the house in a gothic style, and replaced the formal gardens with an informal natural landscape, thus beginning the evolution of the grounds towards their current form.
The estate passed through the hands of two more members of the Cavendish family until in 1834 it was inherited by William Cavendish, 2nd Earl of Burlington. The Earl reconstructed the house between 1838 and 1842 with the assistance of the local architect Webster of Kendal. It acquired a gothic appearance throughout, with the addition of ornamental chimneys, gables and mullioned and transomed windows. The gardens were also enlarged at this time. In 1858 the Earl became the head of the Cavendish family, inheriting the title of Duke of Devonshire, along with Chatsworth House, which is one of the grandest houses in England, and several other residences. However, he continued to spend much time at Holker, partly because it was near to the town of Barrow-in-Furness, which he largely owned, and where, in an unusual move for a Victorian landowner, he made vast (and ultimately unsuccessful) investments in heavy industry.
In 1871 the west wing of the house, containing the main family apartments, was destroyed by fire. Most of the contents were lost, including many of the fine paintings collected by the second Sir William Lowther in the mid 18th century. The Duke had it rebuilt to much the same plan, but in a grander style. It was described by Sir Nikolaus Pevsner: as "the grandest of its date in Lancashire, and it is moreover by the best architects living in the county Paley and Austin. It is their outstanding work, red sandstone in the Elizabethan style." The interiors are comfortably domestic, well lit and airy, with much wood panelling; a relaxed family retreat built for a man whose main country and London houses were opulent showplaces. The whole of the two principal floors of this wing are now open to the public, while the family lives in the older part of the house. Holker later passed to Lord Richard Cavendish, a younger brother of the 9th Duke of Devonshire, whose descendants including Hugh Cavendish still live there.
Holker Hall's 25 acre garden is a fine example of a mixed English garden, with formal and informal elements. The plantsmanship is of the first order. Features include an arboretum, herbaceous borders, a Victorian rockery and a late 20th century cascade. The garden houses the National Collection of styracaceae. In 1991 Holker won the Historic Houses Association Garden of the Year Award and in 2002, Holker's Great Lime, which probably dates from the early 17th century, was selected as one of Britain's 50 Great Trees in honour of Queen Elizabeth II's Golden Jubilee. The garden blends into the surrounding park. The current Lady Cavendish, Grania, has developed the garden and also the visitor facilities at the house.
The Lakeland Motor Museum is located at Holker Hall.