Within 20 years top hats had become popular with all social classes, with even workmen wearing them. At that time those worn by members of the upper classes were usually made of felted beaver fur, while those worn by working men were made of rabbit fur; the generic name "stuff hat" was applied to hats made from fur. The hats became part of the uniforms worn by policemen (who could stand on them to look over walls) and postmen (to give them the appearance of authority); since these people spent most of their time outdoors, their hats were topped with black oilcloth.
During the early part of the 19th century felted beaver fur was gradually replaced by silk "hatter's plush", though the silk topper met with resistance from those who preferred the beaver hat. A short-lived fad in the 1820s and 1830s was the "Wellington" style of top-hat with concave sides. The peak of the top hat's popularity in the 1840s and the 1850s saw it reach its most extreme form, with ever higher crowns and narrow brims. The stovepipe hat was a variety with straight sides, while one with slightly convex sides was called the "chimney pot". The stovepipe hat was popularized in the US by Abraham Lincoln during his presidency; it is said that Lincoln would keep important letters inside the hat.
During the middle part of the 19th century the top hat developed from a fashion into a symbol of urban respectability, and this was assured when Prince Albert started wearing them in 1850; the subsequent rise in popularity of the top hat led to a decline in beaver hats, sharply reducing the size of the beaver-trapping industry in North America.
The nineteenth century is sometimes known as the Century of the Top Hat. The historian James Laver once made the observation that an assemblage of "toppers" looked like factory chimneys and thus added to the mood of the industrial era. In England, post-Brummel dandies went in for flared crowns and swooping brims. Their counterparts in France, known as the “Incroyables,” wore top hats of such outlandish dimensions that there was no room for them in overcrowded cloakrooms until Antoine Gibus came along in 1823 and invented the collapsible top hat. Such hats are often called an "opera hat", though the term can also be synonymous with any top hat, or any tall formal men's hat. In the 1920s they were also often called "high hats".
Men wore top hats for business, pleasure and formal occasions — pearl gray for daytime, black for day or night. At one point Top hat etiquette dictated a man should not wear it flat on his head. He should wear it tilted forward and to one side — very slightly though, no more than 10 degrees in either direction — about the same angle Lord Ribblesdale wore his in the famous portrait by John Singer Sargent.
However, at its peak in popularity a reaction developed against the top hat, with the middle classes adopting bowler hats and soft felt hats such as fedoras, which were more convenient for city life, as well as being suitable for mass production. In comparison, a top hat needed to be handmade by a skilled hatter, with few young people willing to take up what was obviously a dying trade. The top hat became associated with the upper class, becoming a target for satirists and social critics. By the end of World War I it had become a rarity in everyday life. It continued to be used for formal wear, with a Morning dress in the daytime and with evening clothes (tailcoat) until the late 1930s. (The top hat is featured as one of the original tokens in the board game Monopoly.)
The top hat persisted in certain areas, such as politics and international diplomacy, for several more years. In the newly-formed Soviet Union, there was a fierce debate as to whether its diplomats should follow the international conventions and wear a top hat, with the pro-toppers winning the vote by a large majority.
Top-hats are sometimes associated with stage magic. In 1814 a French magician named Louis Comte became the first conjurer on record to pull a white rabbit out of a top hat. They also appear as a form of party hat and are popular amongst persons in the gothic subculture.