Oliver was the son of Raymond de Termes, count of the Termes region of the Carcassonne district, then lying on the border between French Languedoc and Spanish Catalonia. The region was overwhelmingly Cathar in religion, and Oliver's father was besieged by Simon de Montfort in his castle at Chateau de Termes in 1210. Having survived the fall of the castle, Oliver made his way south with other refugees from the crusade to the court of Aragon, where he befriended the future King James I of Aragon. He was also companions in youth with Raymond Trencavel of Albi, and Raymond the son of the dispossessed Count of Toulouse.
Oliver fought first against the knights who had conquered his country during the crusade against the Cathars, then in about 1245 he made peace with King Louis IX, agreeing to go on crusade. He spent much of the remainder of his life assisting in military operations in the Holy Land (participating in both the 7th and 8th Crusades), advising the king, and helping pacify Languedoc.