Originally this was a Norman castle held by the Bigod family. It was confiscated by Henry II in 1157 but restored to the Bigods around 1164. The castle was rebuilt by Hugh Bigod in 1165. It was again confiscated following the 1173-4 revolt and the great tower was pulled down. However it was restored yet again to the Bigods and was further developed in 1294 by Roger Bigod.