A three storey tower was built by Sir William Caxton on the site of an earlier pele tower in the 15th century.
In 1621 the castle was acquired by the Radclyffe family as a result of the marriage of Edward Radclyffe to the Dilston heiress. The Catholic Radclyffes built a private chapel adjacent to the house in 1616 (the chapel also has Ancient Monument and Listed Building status).
In 1622 Sir Francis Radclyffe incorporated the tower house into a new manor house, which was to become known as Dilston Hall.
A later Francis Radclyffe was a supporter of the Royalist cause during the Civil War and his estates including Dilston Hall were sequestered by the Commonwealth. The property was restored to the 3rd Earl who in 1709 began an ambitious programme to replace the old house with a substantial mansion. The new mansion was never completed. The 3rd Earl took part in the Jacobite uprising of 1715, was convicted of treason and executed in 1716.
His brother Charles, also involved in the rebellion, escaped to France. He returned to support the later 1745 uprising, was captured and executed in 1746. The Radclyffe estates were forfeited to the Crown by Act of Parliament (22 George II Chapter 52) and in 1748 Dilston Hall and lands were granted to the Greenwich Royal Hospital.
The Hall was demolished in 1765 leaving standing only the castle tower and the chapel.
A restoration of the buildings began in 2001 and the castle was opened to the public in 2003