According to the United States Census Bureau, the town has a total area of 4.4 square miles (11.3 km²), of which, 3.7 square miles (9.5 km²) of it is land and 0.7 square miles (1.8 km²) of it (16.06%) is water.
There were 536 households out of which 21.1% had children under the age of 18 living with them, 30.4% were married couples living together, 19.6% had a female householder with no husband present, and 48.1% were non-families. 43.7% of all households were made up of individuals and 26.3% had someone living alone who was 65 years of age or older. The average household size was 2.11 and the average family size was 2.91.
In the town the population was spread out with 22.1% under the age of 18, 7.1% from 18 to 24, 22.0% from 25 to 44, 25.0% from 45 to 64, and 23.8% who were 65 years of age or older. The median age was 44 years. For every 100 females there were 76.4 males. For every 100 females age 18 and over, there were 75.9 males.
The median income for a household in the town was $22,237, and the median income for a family was $29,167. Males had a median income of $25,536 versus $23,984 for females. The per capita income for the town was $13,789. About 21.5% of families and 28.4% of the population were below the poverty line, including 43.4% of those under age 18 and 23.0% of those age 65 or over.
Cape Charles was, for many years, the terminal for passenger and car ferry service across the bay to Norfolk and Hampton. The last ferry left Cape Charles in 1953. Cape Charles still serves as a terminal for railway barges that carry rail cars from the former Eastern Shore Railroad which is now Bay Coast Railroad across the mouth of the Bay to Norfolk.
The town hosted the Northampton Red Sox in the old Eastern Shore Baseball League.