It takes its name from a house built in the 1670s by Patrick Fea, a retired privateer, whose descendant John Fea pioneered the kelp burning industry on the island in 1722.
The fall of the herring industry came with overfishing, and the outbreak of WWII. There was also a similar decline during WWI, but a brief rebirth in the interbellum. Until then it had been the biggest herring port in the northern Orkneys, trading mainly with the Baltic ports. At its height, 300 boats were moored in Whitehall harbour, along with fifteen curing stations, and 1500 fish wives. Many of the population were itinerant. As a side effect, there were forty pubs here in the high season, and many houses took lodgers. The wealth can be seen in the large houses which still dominate the town.
Aside from herring fishing and kelp burning, there was also a smaller industry in occasional whaling. Fifty whales were once driven into nearby Mill Bay for oil, in a move similar to the Faroese grindadráp.