There are several competing folk derivations of this phrase, but they are regarded by philology as somewhat far-fetched.
One derivation holds that it is a reference to the Danish poll tax on the Irish in the ninth century. The story goes that the Danes slit the noses of non-payers. However, the phrase did not appear in print until 1672.
Another possible explanation is placed on Viking raiders. Vikings required the payment of tribute from cities they did not raid. If the king refused to pay tribute, they would slit their noses. Hence, the king paid "through" the nose.
Another explanation relies on the use of rhino as a slang for money. Rhinos is the Greek word for nose, thus a rather loose connection is supposed to be set up between paying and noses. A nosebleed is a metaphor for being "bled dry" of money.
In sixteenth-century English, Italian, Greek and Latin, however, there is a well-established expression, "to lead by the nose," which means to force or control someone's actions, as a farmer would lead a bull by a ring through its nose. It also has connotations of making a fool of someone. ('led by the nose, as asses are' (Othello, Act I, Sc 3). Thus, to pay through the nose could have originated as an extension of this, denoting payment for something at a premium because of a lack of alternative options.