In the 1600s the island was granted to William Brewster of Plymouth Colony, after whom the adjacent islands of Great Brewster, Little Brewster, Middle Brewster and Outer Brewster are named. Subsequently it was owned by Charles Apthorp who also owned Long Island and other property in the harbor. Over the years, several huts and houses were built on the island, including a colonial style summer estate built by Benjamin P. Cheney, Jr. and his wife, actress Julia Arthur, in 1902. Today only ruins remain. Illegal boxing matches were staged on the island in the late 19th century.