A fifth of liquor was equal to one-fifth of a gallon, which is a common unit of measurement for distilled spirits in the United States. Even though people colloquially refer to liquor bottles as "fifths," since 1980, they have actually been 750 milliliters or 25.36 ounces, which is slightly less than exactly one-fifth of a gallon.
In the late 1800s, distilled alcohol in the U.S. was sold in quart bottles that were actually filled with slightly less than a quart, which is 32 ounces. This may have been in part because selling alcohol in quantities greater than one-fifth of a gallon would have qualified a seller as selling alcohol at the wholesale level. A gallon is 128 ounces, and until 1980, the fifth, which is exactly 25.6 ounces, was the standard bottle size for distilled liquor.
Eventually, efforts in the U.S. toward adopting the metric system that had taken shape in the 1970s led to changes implemented by the Bureau of Alcohol, Tobacco and Firearms, which is the federal agency that regulates alcohol sales. Bottle measurements were standardized, and what was commonly known as the fifth became 750 milliliters, which was, and still is, the standard size of a bottle of wine in Europe and elsewhere.