The bay itself is quite small, but well sheltered, with a small bark area and an extended coastline stretching out to the north. Nearby John F. Kennedy Park contains old WWII bunkers that were used to survey the Hauraki Gulf. The northern headland of Castor Bay is the site of an old Maori Pa, Rahopara, that is believed to have been last inhabited and then abandoned in the 18th century, prior to European colonisation. Very little is known about the people who lived there but the tribe's ancient earth excavations are still visible.

Sam Hunt, one of New Zealand's most well known poets was born in Castor Bay.

The local primary school is Campbells Bay Primary School located on Aberdeen Road, nearby the Pupuke Golf Course.