During this period of his life, he acquired a reputation principally as a free-spending, gambling, sporting man. He married Lady Frances Seymour (1728–1761), daughter of Charles Seymour, 6th Duke of Somerset on 3 September 1750. They had six children:
In 1752, the Pelham ministry suggested to George II that Granby be appointed colonel of the prestigious Royal Horse Guards (Blues), in order to secure the parliamentary support of his family. The king denounced Granby as a sot and a bully, and initially refused to make the appointment. In the meantime, Granby advanced his parliamentary career, and was returned for Cambridgeshire in 1754. Though he despised faction in government, he allied himself with Viscount Royston, the other knight of the shire, a government whig. The king came to view him more favorably, and he defended the Newcastle ministry in the House of Commons. He was promoted major-general in 1755, and was at last made Colonel of the Blues in 1758.
In the remaining campaigns of the Seven Years' War the English contingent was more conspicuous by its conduct than the Prussians themselves. On 31 July 1760 Granby brilliantly stormed Warburg at the head of the British cavalry, capturing 1500 men and ten pieces of artillery. Since his twenties, he had been almost entirely bald, but disdained to wear a wig; during the charge, he lost his hat, giving rise to the expression "going at it bald-headed". A year later (15 July 1761) the British defended the heights of Villinghausen with what Ferdinand himself styled "indescribable bravery". On the following day, he led his troops in a counter-attack and helped drive the French from the field. His opponent, the duc de Broglie, was so impressed that he commissioned a portrait of Granby by Sir Joshua Reynolds. In the last campaign, at Gravenstein and Wilhelmsthal, Homberg, Gudensberg and Cassel, Granby's men bore the brunt of the fighting and earned the greatest share of the glory. Lord Ligonier praised his conduct at Wilhelmsthal, where he cut off the French rearguard as "Il a manoeuvré comme un ange … no man ever acted with more courage or more like a commanding officer." His last field action was at Brückermühl, where he brought two brigades to the relief of General Zastrow.
Granby supported the government's issue of general warrants and prosecution of Wilkes, but in 1765 spoke against the dismissal of army officers for voting against the government in Parliament. In May 1765, Lord Halifax attempted to persuade George III to appoint Granby Commander-in-Chief of the Forces, in the hopes that his popularity would help quell the riot of the London silk weavers. The king refused, having promised the reversion of the post to the Duke of Cumberland, but obtained Granby's retention as Master-General of the Ordnance in the new Rockingham ministry, although Granby did not co-operate with the ministry and voted against the repeal of the Stamp Act.
Under the Chatham Ministry, Granby was appointed commander-in-chief on 13 August 1766. Despite rumors of his retirement, he vigorously electioneered during the 1768 season, and increased the Rutland interests seats to seven, at some expense. With the resignation of Chatham, he found himself somewhat isolated in the Grafton Ministry. While he had opposed the attempts of the government to expel Wilkes from his seat in Middlesex, his personal dislike of Wilkes overcame his principles, and he voted in favor of the expulsion on 3 February 1769 and for the seating of Henry Luttrell afterwards. It was to prove a capital political mistake. Junius had written his first attack on the ministry at the end of January, not excluding Granby, whom he condemned for servility towards the court and personal corruption. Granby's great popularity might have let him ride out the affair, but his reversal on Wilkes provided new ammunition. Worse still, a reply to Junius by his friend Sir William Draper, intended in his defense, essentially validated the charge that the hard-drinking and personable Granby was easily imposed upon by less scrupulous acquaintances.
Ultimately, it was not the attacks of Junius, but the return of Chatham that brought about his departure from politics. Granby had always respected Chatham, and through the intermediation of John Calcraft, was eventually persuaded to break with the ministry. On 9 January 1770, he announced that he had reversed himself once more on the propriety of expelling Wilkes, and shortly thereafter resigned as commander-in-chief and master-general of the ordnance, retaining only the colonelcy of the Blues.
Once out of office, Granby found himself hard-pressed by his creditors, and the loss of his official salaries had weakened his financial position. In the summer of 1770, he unsuccessfully campaigned for George Cockburne at the Scarborough by-election. He remained to recuperate from illness, but died on 18 October 1770 from a seizure precipitated by "gout in the stomach". He had been made a privy councillor in 1760, Lord Lieutenant of Derbyshire in 1764, and LL.D. of Cambridge in 1769.
Granby was a popular subject for portraits: Sir Joshua Reynolds and his studio painted him at least 16 times. In these portraits Reynolds shows him hatless and wigless, as Granby wanted to be seen — as the bald-headed general who really went at it and fought hard among private soldiers. The portrait commissioned by de Broglie, his admiring opponent at Villinghausen, shows him in the uniform of the Blues and leaning on his horse. This painting is now in the John and Mable Ringling Museum of Art, Florida, and an autograph copy is in the British Royal Collection. Another Reynolds portrait of Granby, in a similar pose but dressed as Master-General of the Ordnance and leaning on a mortar, is in the National Army Museum, London.
Granby's contemporary popularity is indicated by the number of inns and public houses which took his name and had his portrait as sign-board. Reputedly, this was in part due to his personal sponsorship of his disabled non-commissioned officers as publicans.