He was the son of Charles d'Albret and Marie de Sully. His father died at the Battle of Agincourt, leaving the younger Charles as lord of Albret and titular Count of Dreux, titular count since after Agincourt the lands of the county of Dreux were in English hands. As a member of the Armagnac faction, Charles was a supporter of the Dauphin Charles, future Charles VII of France. His links to the Armagnacs were strengthened by his marriage in 1417 to Anne of Armagnac, daughter of Bernard VII, Count of Armagnac, from whom the faction took its name. The children of Charles and Anne included Jeanne, who would marry the Constable of France, Arthur de Richemont, and Jean, from whom King Jean d'Albret and Queen Jeanne d'Albret, titular rulers of Navarre, were descended.
Charles served on the royal council of Dauphin Charles and in 1427 joined with Richemont and Yolande of Aragon in removing the Dauphin's unpopular favourite Pierre de Giac. De Giac was soon afterwards executed by drowning. He took part in the campaigns of Joan of Arc, and was named lieutenant of the province of Berry. He was confirmed in possession of the county of Dreux in 1441 by King Charles VII.
On his death Albret passed to his grandson Alain (called Alain the Great), Jean's son, but he left the county of Dreux to his third son, Armand-Amanieu. Alain, however, seized control of the county. His fourth son, Charles, was executed for treason in 1472.