It was built to commemorate Commodore Sir William James who, in April 1755, attacked and destroyed the island fortress of Suvarnadurg (rendered in English: Severndroog) along the Bombay and Goa coast of India. He died in 1783 and the castle was built as a memorial to him by his wife, Lady James of Eltham.
A Grade II listed building, the Gothic-style castle is 63 feet (19 m) high and triangular in section, with a hexagonal turret at each corner. From its elevated position, it offers views across London, with features in seven different counties visible on a clear day.
In 1988, the local council could no longer afford the building's upkeep and it was boarded up. In 2002, a community group, the Severndroog Castle Building Preservation Trust, was established. In 2004, it featured in the BBC TV series Restoration (presented by Griff Rhys Jones, Ptolemy Dean and Marianne Suhr, producer-director Paul Coueslant) - with the aim of gaining support for a programme of work to restore the building and open it to the public.