He was born in Paris, the son of Charles-Louis-Victor, prince de Broglie and grandson of Victor-François, 2nd duc de Broglie. While his grandfather emigrated, his parents were imprisoned during the Terror. His father was guillotined in 1794, but his mother managed to escape to Switzerland, where she remained until the fall of Robespierre. She then returned to Paris with her children and lived there quietly until 1796, when she married the Marc-René-Voyer de Paulmy, marquis d'Argenson, grandson of Louis XV's minister of war. On his grandfather's death in 1804, Victor de Broglie became the third duc de Broglie. Under the care of his stepfather, the young duke received a careful and liberal education and made his entrée into the aristocratic and literary society of Paris under the First French Empire. In 1821, his wife, the daughter of Erik Magnus Staël von Holstein (himself husband of Madame de Staël), gave birth to Albert, who would become the fourth duke of Broglie.
After this defiant act of opposition it was perhaps fortunate that his impending marriage gave him an excuse for leaving the country. On 15 February 1816, he was married at Leghorn to Albertine, baroness Staël von Holstein, the daughter of Madame de Staël. He returned to Paris at the end of the year, but took no part in politics until the elections of September 1816 broke the power of the ultraroyalists and substituted for the Chambre introuvable a moderate assembly composed of liberal Doctrinaires. Broglie's political attitude during the years that followed is best summed up in his own words:
From 1812 to 1822 all the efforts of men of sense and character were directed to reconciling the Restoration and the Revolution, the old régime and the new France. From 1822 to 1827 all their efforts were directed to resisting the growing power of the counter-revolution. From 1827 to 1830 all their efforts aimed at moderating and regulating the reaction in a contrary sense.
During the critical time that followed, he consistently supported the principles which triumphed with the fall of Laffitte, representant of the center-left Parti du mouvement, and the accession to power of Casimir Perier, leader of the center-right Parti de la résistance, in March 1831. After the death of the latter and the insurrection of June 1832, Broglie took office once more as Minister for Foreign Affairs (11 October).
His tenure of the foreign office was coincident with a very critical period in international relations. But for the sympathy of Britain under Palmerston, the July Monarchy would have been completely isolated in Europe, and this sympathy the aggressive policy of France in Belgium and on the Mediterranean coast of Africa had been in danger of alienating. The Belgian crisis had been settled, so far as the two powers were concerned, before Broglie took office, but the concerted military and naval action for the coercion of the Dutch, which led to the French occupation of Antwerp, was carried out under his auspices. The good understanding of which this was the symbol characterized also the relations of Broglie and Palmerston during the crisis of the first war of Muhammad Ali with the Porte, and in the affairs of the Spanish peninsula their common sympathy with constitutional liberty led to an agreement for common action, which took shape in the Quadruple Alliance between Britain, France, Spain, and Portugal, signed at London on 22 April 1834. Broglie had retired from office in the March preceding, and did not return to power until March of the following year, when he became head of the cabinet.
One of Broglie's first act on his return was to have the National Assembly ratify the 4 July 1831 treaty with the United States, which it had rejected during his first term. His cabinet also voted the 1835 laws restricting freedom of press, following Giuseppe Fieschi's attempted assassination against Louis-Philippe in July 1835.
In 1836, the government having been defeated on a proposal to reduce the five percents tax, he once more resigned.
From 1836 to 1848 de Broglie held almost completely aloof from politics, to which his scholarly temperament little inclined him, a disinclination strengthened by the death of his wife on 22 September 1838. His friendship for Guizot, however, induced him to accept a temporary mission in 1845, and in 1847 to go as French ambassador to London.
The revolution of 1848 was a great blow to him, for he realized that it meant the final ruin of the constitutional monarchy, in his view the political system best suited to France. He took his seat, however, in the republican National Assembly and in the Convention of 1848, and, as a member of the section known as the "Burgraves", fought against both socialism and what he foresaw as a coming autocratic reaction. He shared with his colleagues the indignity of the 2 December 1851 coup, and remained for the remainder of his life one of the bitterest enemies of the Second Empire, though he was heard to remark, with that caustic wit for which he was famous, that the empire was the government which the poorer classes in France desired and the rich deserved .
The last twenty years of his life were devoted chiefly to philosophical and literary pursuits. Having been brought up by his stepfather in the sceptical opinions of the time, he gradually arrived at a sincere belief in the Christian religion. "I shall die," he said, "a penitent Christian and an impenitent Liberal."
His literary works, though few of them have been published, were rewarded in 1856 by a seat in the Académie française, replacing Louis de Beaupoil, comte de Sainte-Aulaire, and he was also a member of the Académie des sciences morales et politiques. In the labors of those learned bodies he took an active and assiduous part.