It was a suitable site, as ships were often becalmed beneath it, when making for Sandy Point Road. It was named after King Charles II, who gave £500 to assist the planters.
Charles Fort was a military post from 1670 until it was abandoned in 1854. Some forty years later in 1890 it was used as a Hansen Home (leper asylum). The Home was closed in 1996.
The fort today is not in good shape. It has been completely abandoned and is overgrown with vegetation.