He was married five or six times. His first wife, Ermessende, Countess of Melgueil, whom he married in 1172, died in 1176 without issue. His second wife was Beatrice of Béziers, sister of Roger II Trencavel; they divorced in 1189 and she was said to have become a Cathar parfaite after the divorce. Raymond and Beatrice had one daughter, Constance of Toulouse, who was married firstly to King Sancho VII of Navarre, and secondly to Pierre-Bermond II of Sauve, lord of Anduze.
Raymond then married for a third time, to Bourgogne, daughter of King Amalric II of Jerusalem and his first wife Eschiva of Ibelin, daughter of Baldwin of Ibelin. He divorced her in 1194. In October 1197 at Rouen he married Joan Plantagenet, but she fled from him in 1199 and died in childbirth. Their only surviving child was Raymond VII of Toulouse (1197-1249).
In Toulouse, he maintained the communal freedoms, extended exemptions from taxation, and extended his protection to the communal territory. A poet and a man of culture, he hated war but did not lack energy, as shown by his dispute with the papal legate Pierre de Castelnau, representative of Pope Innocent III. Pierre's assassination on January 15, 1208 led to Raymond's excommunication. The excommunication was lifted after Raymond humbled himself before the Pope.
After the capture and massacre of Béziers, the siege and capture of Carcassonne, and the death of Raymond-Roger of Trencavel, he moved his camp, was again excommunicated by the Council of Montpellier in 1211, and tried to organize resistance against the Albigensian Crusade. More of a diplomat than a soldier, he was unable to stop of the advance of Simon de Montfort, who conquered Toulouse. Raymond was exiled to England under his former brother-in-law John Plantagenet.
In November 1215 Raymond and his son (the later Raymond VII of Toulouse) were in Rome with Raymond-Roger of Foix on the occasion of the Fourth Lateran Council) to vindicate themselves and dispute the loss of their territories. Raymond's son-in-law, Pierre-Bermond II of Sauve, was also there to lay claim to the county of Toulouse, but this claim failed. Raymond and his son went from Rome to Genoa and thence to Marseille in February 1216. Raymond's son set out from Marseille to regain the family territories in Provence; in May 1216 he besieged Beaucaire and captured it on August 24.
Meanwhile Raymond went to Aragon, hoping to rally support. From there he engaged in secret negotiations with leaders in Toulouse during 1216. Simon de Montfort possibly believed that Raymond was on his way to the city in September 1216; at any rate he returned in great haste from Beaucaire and conducted a partial sack of the city, apparently intended as punishment. Finally, on September 12, 1217, Raymond re-entered Toulouse again. Simon de Montfort immediately besieged the city once more. Simon was killed during the siege (on 25 June 1218); his son Amaury VI of Montfort took his place, and for five years the Crusade faltered. The failure of Louis VIII's campaigns, from 1219 to 1226, finally permitted Raymond, and his son and successor, to recover most of their territories.