Like other inner city residential areas, Upper Manhattan could be described as the "non-tourist" section of Manhattan. Until the late 20th century it was less influenced by the gentrification that had taken place in other parts of New York over the previous 30 years. Not only do New York tourist maps not normally acknowledge the outer boroughs of the Bronx, Brooklyn, Queens, and Staten Island (and if so, only in the most rudimentary way), but they also regularly neglect Upper Manhattan as well. This is likely due to perceptions that it is less glitzy, less tourist-friendly, less distinctively New York, and more crime-ridden. Even many native New Yorkers tend to treat Upper Manhattan like an outer borough, its distance from Midtown Manhattan and comparatively lower rents leading many to exclude this northern neck from the area New Yorkers call "The City".
The Upper West Side is more affluent than its northern neighbors such as Manhattanville, Morningside Heights, Inwood, etc. The Upper East side is similarly more affluent than its northern neighbor Spanish Harlem. Thus the 96th street definition comes from the East Side as well. Gentrification not yet completed in some neighboorhoods, crime, graffiti, etc are associated with Upper Manhattan. Also, tourist attractions and the like are concentrated mainly outside Upper Manhattan, while attractions do exist in Upper Manhattan the popular iconic landmarks are located in lower Manhattan. Examples of such include skyscrapers, theater and Broadway shows, prime restaurants (liquor laws have restricted growth of restaurants in upper Manhattan), clean streets, etc link title Gentrification is occurring at a different pace, rate, and style than Soho, East Village, etc due to different demographics.
All of Upper Manhattan is contained in the larger area New Yorkers know as Uptown (above 59th Street.) The Bronx, though not in Manhattan, is often colloquially referred to as "Uptown", especially in the context of hip hop/inner-city culture.