An engineer by profession, Wharton took up refereeing at the age of 21 and had reached Class I status within 3 years. He refereed a number of important and high profile games during his career, including the Scottish Cup finals of 1961-62, 1962-63, 1965-66 and 1970-71, and the League Cup finals of 1960-61, 1962-63, 1966-67 and 1970-71.
Wharton's prowess was recognised outside Scotland as well and he officiated 16 international matches, from Belfast to Brazil. He also oversaw 24 international club fixtures, including the 1962 Cup Winners Cup Final between Atlético Madrid and Fiorentina at Hampden Park.
In later years, Wharton was chairman of the Scottish Football Association's Referee Supervisors Committee between 1976 and 1990 and served as part of FIFA's Referees Committee between 1981 and 2000. He was awarded the OBE in 1990 for his services to Scottish football and FIFA's Order of Merit in Gold in 1992. He was described by FIFA president Sepp Blatter as "one of the world's most distinguished refereeing officials".