It was built by the Peverel family in the 12th century and became Crown property in 1155 when the third William Peverel fled into exile. A stone keep was built around 1173, surrounded by a curtain wall which was eventually breached in 1216 during the reign of King John. It was then allowed to deteriorate into a ruin. Surviving fragments of the curtain wall were later incorporated in a wall walk that can be seen in the castle garden.
The manor and castle were purchased by Sir George Talbot in 1553. They were sold by Gilbert, 7th Earl of Shrewsbury to Sir Charles Cavendish, son of Bess of Hardwick in 1608. Sir Charles set about re-building the castle, a process which was continued by his son William Cavendish, later 1st Duke of Newcastle upon Tyne. Despite its embattled appearance, it was designed for elegant living rather than for defence. The tower, known today as the 'Little Castle', was completed around 1621.
During the Civil War Bolsover Castle was taken by the Parliamentarians who slighted it and it again fell into a ruinous state. However William Cavendish added a new hall and staterooms to the Terrace Range and, by the time of his death in 1676, the castle had been restored to good order.
It passed through the female line into the Bentinck family, and ultimately became one of the seats of the Dukes of Portland. After 1883 the castle was uninhabited and eventually given to the nation by the 7th Duke of Portland in 1945. The castle is now in the care of English Heritage.