Swainston Manor is a mile to the east of Calbourne. Now a hotel, Swainston Manor was originally a manor house on a site dating back to 735 CE. Eight hundred years ago it became the location of a palace built by the Bishops of Winchester. It has a 12th century chapel on its 32 acres. Most of the present building was constructed in the 1700s, but an attached hall dates from the 1200s. Warwick the Kingmaker reportedly dined at Swainston Manor.
The parish of Calbourne was at one time dependent on Swainston Manor.
The house was severely damaged by incendiary bombs during World War II. The church on the site however was untouched.
The poet Alfred Lord Tennyson also visited Swainston a few times. It is claimed that he wrote "Maud" on its grounds. He also wrote "In the Garden at Swainston" after the death of his friend and Swainston's owner, Sir John Simeon.
Swainstone was one of the locations for King James 1st public school during the 1960s. Other locations for the school included Newport and Cowes.