The city of New Haven and its neighborhoods of City Point, Long Wharf, The Annex, and East Shore lie on the northern and eastern sides. West Haven is on the west. The Quinnipiac and Mill rivers converge and empty into the inlet on its north end. The Pearl Harbor Memorial Bridge crosses here. The West River empties into the western end of the harbor (also known as West Haven Harbor).
The harbor is protected from Long Island Sound by a peninsula from its western side, once known as "Little Necke" but now called Lighthouse Point, because of the lighthouse that was constructed on its tip in 1805. The original lighthouse was replaced in 1845 by the current structure, called the Five Mile Point Lighthouse. This lighthouse was replaced for navigation in 1877 by the offshore Southwest Ledge Light. Sperry Lighthouse (1899-1933) also served the harbor.
The harbor is circumnavigable via the partially-completed "Harborside Greenway" bicycle and pedestrian trail, which is part of the East Coast Greenway system.