The first settler of the island, William Black, aka Black Will, a freed slave from Kittery, Maine, took possession of the island in 1727 after spending many years of his life there. Because of this, the island became known as Will's Island.
In 1742, Reverend Timothy Bailey may have bought Will's Island for one pound of tobacco and a gallon of rum from William Black. Another variation of this history is that the minister's wife liked the island and the Bailey's bribed municipal officials to find some flaw in Will's title and award the island to them. In any event, after Timothy Bailey and his wife took possession of the island, William Black left to live on Orr's Island. From then on, the island was known as Bailey Island.
The Bailey Island Bridge, which spans Will's Gut and connects Bailey Island to Orr's Island, was completed in 1928. Will's Gut is the only geographical feature that still bears the name of the original non-native inhabitant of Bailey Island.
One popular rumor that has persisted for at least the last half century is that the bridge from Orr's Island to Bailey Island was one of two granite cribstone bridges, ever built in the world, and that the other one in Scotland, collapsed shortly after construction. In fact, it appears to be the only such bridge ever constructed. The unique construction of the bridge permits the substantial tides of that area to flow freely through it, greatly reducing the effect that flow would otherwise have on boats transiting its narrow channel opening.
Three prominent female psychoanalysts, Esther Harding, Eleanor Bertine, and Kristine Mann, were followers of Carl Jung and long-time summer residents of Bailey Island. Kristine Mann , known in Jung's work as Miss-X, was one of Jung's subjects. Jung is also known to have visited Bailey Island and to have lectured at the Bailey Island Library Hall.
Frank Aydelotte, president of Swarthmore College, summered at Bailey Island where he became acquainted with Holbrook Mann MacNeille and Stephan Mann MacNeille in the 1920s. With Aydelotte's encouragement both attended Swarthmore and later went on to lead distinguished careers in physics and mathematics.