The neighborhood took form by about 1820 next to the site of the former Collect Pond, which had been drained due to a severe pollution problem. The landfill job on the Collect was a poor one, and surface seepage to the southeast created swampy, insect-ridden conditions resulting in a precipitous drop in land value. Most middle-class-and-up inhabitants fled, leaving the neighborhood completely open to the influx of poor immigrants that started in the early 1820s and reached a torrent in the 1840s due to the Irish Potato Famine. It was situated close enough for a walking commute to the large mercantile employers of the day in and around the dockyards at the island’s southern tip, but it was far enough away from the built-up Wall St. area to allow a total remake of character.
At Five Points’ “height”, only certain areas of London’s East End vied with it in the western world for sheer population density, disease, infant and child mortality, unemployment, prostitution, violent crime, and other classic ills of the destitute. However, it was the original American melting pot, at first consisting primarily of newly emancipated African Americans (gradual emancipation ended in New York on July 4, 1827), and newly arrived Irish.
The rough and tumble local politics of “the ould Sixth ward” (The Points’ primary municipal voting district), while not free of corruption, set important precedents for the election of non-Anglo-Saxons to key offices. Although the tensions between the African Americans and the Irish were legendary, their cohabitation in Five Points was the first large-scale instance of volitional racial integration in American history. In the end, the Five Points African American community moved to Manhattan’s West Side and to the then-undeveloped north of the island.
Five Points was dominated by rival gangs like the Roach Guards, Dead Rabbits, and Bowery Boys. According to Herbert Asbury's book "The Gangs of New York," police arrested 82,072 New Yorkers in 1862, or 10% of the city. In 1864, five police officers were murdered. To give a sense of the era, Asbury's book tells the story of a little girl who lived with 25 people in a small basement room, and was stabbed to death for a penny she had begged. Asbury reports that the girl's body lay in a corner for five days before her mother dug her a shallow grave in the floor.
In the twentieth century, the Five Points Gang recruited members from the toughest gangs in the city. Five Points mobsters included Paul Kelly, Giovanni "Johnny" Torrio and Frankie Yale. Recruits included Charles "Lucky" Luciano and Al Capone. Capone received his nickname "scarface" from a knife fight at The Harvard Inn, in Governor's Island. When Capone was finally convicted for tax evasion in 1931, he was quoted in newspapers saying, "I shoulda never left Five Points." (meaning his New York gang of that name).
Almack’s dance hall (also known as “Pete Williams’s Place”) on the east side of Orange St. (today’s Baxter St.), just south of its intersection with Bayard St., was home to a fusion of Irish reels and jigs with the African shuffle. This music and dance had spontaneously appeared on the street from competition between African-American and Irish-American musicians and dancers, spilling into Almack's where it gave rise in the short term to Tap Dance (see Master Juba) and in the long term to a music hall genre that was a major precursor to American Jazz and Rock and Roll. This ground is today occupied by Columbus Park, used primarily by residents of modern Chinatown.
The most enduring description of the neighborhood was penned by Charles Dickens in his 1842 work American Notes. As he strolled about Manhattan in his first visit to the United States he did not shrink from the worst areas of town. His account of the filth and wretchedness characterizing so much of the Five Points was balanced by an admiring description of the patrons of Almack's.