It was built as a manor house by Sir John de Broughton in 1300 at a location where the confluence of three streams created a natural site for a moated manor. The house was sold in 1377 to William of Wykeham, Bishop of Winchester. The original house was crenellated by Sir Thomas Wykeham in 1406. In 1451 it passed by inheritance to the Fiennes family (Barons Saye and Sele). Beginning in 1550 Richard Fiennes transformed the medieval manor into a house in the Tudor style.
In the 17th century William Fiennes, 1st Viscount Saye and Sele was one of the leading activists against Charles I. He raised troops to fight against the king at the inconclusive Battle of Edgehill in 1642. Royalist troops later besieged the castle, overcoming the defenders and occupying the castle for a time.
It is still the home of the Saye and Sele family.
Parts of the films The Scarlet Pimpernel (1982), Three Men and a Little Lady (1990), The Madness of King George (1994) and Shakespeare in Love (1998) were shot in the castle. TV filming for parts of Elizabeth The Virgin Queen, Friends and Crocodiles,1975 Morecambe and Wise Christmas Show and Star Maidens also took place there.