Phipps was at Eton College with Joseph Banks, but left early to go to sea with his uncle Captain The Hon. A. J. Hervey. In 1766 he sailed to Newfoundland as Lieutenant on HMS Niger. Banks accompanied him as ship's naturalist.
On June 4, 1773 Phipps, now a captain, set off from Deptford on a voyage towards the North Pole. He had two ships, the Racehorse and the Carcass. Phipps took with him Dr Irving as naturalist and doctor, and Israel Lyons (1739–1775) as astronomer. One of the crew aboard Carcass was a young Horatio Nelson. They sailed beyond Svalbard to the Seven Islands, but were forced back by the ice and returned to Orfordness on September 17. During the voyage Phipps was the first European to describe the Polar Bear and the Ivory Gull, which were included in his A Voyage towards the North Pole undertaken ... 1773 (1774).
In 1775, he succeeded his father as Baron Mulgrave (Ireland). He was created Baron Mulgrave in the Peerage of Great Britain in 1790, but that title became extinct on his death in 1792; his brother Henry Phipps succeeded him in the Irish barony.