Foveaux was baptized on 6 April 1767 at Ampthill, Bedfordshire, England, the sixth child of Joseph Foveaux and his wife Elizabeth, née Wheeler. Family tradition maintains he was actually born on 10 April 1766, almost a year earlier.
Foveaux was an ensign in the 60th regiment and then joined the New South Wales Corps in June 1789 as lieutenant and reached Sydney in 1791. There he was promoted to major and, as senior officer between August 1796 and November 1799, he controlled the Corps at a time when the senior officers were making fortunes from trading and extending their lands. He soon became the largest landholder and stock-owner in the colony.
In 1800, having established a reputation as an able and efficient administrator, he offered to go to Norfolk Island as Lieutenant-Governor. Finding the island run down, he built it up with particular attention to public works, for which he earned the praise of Governor King.
During this period, part of the first settlement of Norfolk Island (1788-1814), Norfolk Island was basically a free settlement with convicts making up no more than 10 per cent of the population. While some individuals were sent from Sydney as a means of isolation, the Island was not a place of secondary punishment as it became in the second settlement (1825-1855).
Judgements of Foveaux's career are often clouded by a manuscript purporting to be the recollections of Norfolk Island gaoler Robert Jones. This document is dated 1823, five years after Jones's death. It contains paintings of buildings on Norfolk Island which were not erected until the 1840s. Modern scholarship reveals it to be a forgery from after 1850 which contains no valid evidence on Foveaux's life and career.
In September 1804 Foveaux left Norfolk Island for England to attend to his private affairs and seek relief for the asthma that affected him. Having recovered, he returned to New South Wales to act as Lieutenant-Governor. On arrival in July 1808, he found Governor Bligh under arrest by officers of the New South Wales Corps in the event known as the Rum Rebellion. Foveaux assumed control, stating that he was not favouring either Bligh or the rebels. His control was characterised by a desire for cheap and efficient administration, improvement of public works, and encouragement of small-holders.
Macquarie was impressed with Foveaux's administration and put him forward as Collin's successor as Lieutenant Governor of Tasmania as he could think of no one more fitting and considered that he could not have acted otherwise with regard to Bligh. However, when Foveaux returned to England in 1810 Macquarie's recommendation was put aside. Foveaux was promoted to Inspecting Field Officer in Ireland and in 1814 became a Major-General. He pursued an uneventful military career after that, rising to the rank of Lieutenant-General in 1830. In 1814 he married Ann Sherwin, his partner since 1793, and they had a daughter born in 1801. He died in London on 20 March 1846.
Foveaux Street is named in his honour.