Its first recorded owner was Ralph Fitzmain who held it in 1154. In the late 14th century a pele tower was built. This was extended to include a great hall, and probably a tower-defended courtyard, by John Cartington in 1442 when he was granted a licence to crenellate his home.
In November 1515 Margaret, Queen of Scots, with her baby daughter Margaret stayed here on her journey from Harbottle Castle. Nearly ten years later, Lord Dacre stationed his troops here on a march north to join the Earl of Surrey.
During the Civil War the castle was a major Royalist centre. Sir Edward Widdrington raised 2,000 foot Royalist soldiers and 200 horse and withheld a Parliamentary siege for over two hours in 1648, but the castle was eventually taken and slighted by the enemy.
Despite some demolition, the castle continued to be occupied until finally abandoned in the 1860s. In 1887 Lord Armstrong partially restored the castle in order to prevent its complete disintegration.