Though originally intended for the ministry, his poetic genius (foreshadowed by the prizes he obtained in 1825 and 1828 for poems on Marcos Botzaris and Julia Alpinula respectively) inclined him towards literary studies. He was named professor of literature at Neuchâtel (1830), but before taking up the duties of his post made a visit to Paris, where he completed his education and became associated with Sainte-Beuve, especially from 1837 onwards. He professed history at Lausanne from 1833 to 1846, when he lost his chair in consequence of the religious troubles.
He then went to Paris, where he remained till 1870, earning his bread by various means, but being nearly forgotten in his native land, to which he remained tenderly attached. From 1845 till 1860 (when the magazine was merged in the Bibliothèque universelle) Olivier and his wife wrote in the Revue suisse the Paris letter, which had been started by Sainte-Beuve in 1843, when Olivier became the owner of the periodical. After the war of 1870 he settled down in Switzerland, spending his summers at his beloved Gryon, and died at Geneva on 7 January 1876.
Besides some novels, a semi-poetical work on the Canton of Vaud (2 vols., 1837-1841), and a volume of historical essays entitled Etudes d'histoire nalionale (1842), he published several volumes of poems, Deux Voix (1835), Chansons lointaines (1847) and its continuation Chansons du soir (1867), and Sentiers de montague (Gryon, 1875).
His younger brother, Urbain Olivier, was also a writer.
Life by Eugène Rambert (1877), republished in his Ecrivains de la Suisse romande (1889), and also prefixed to his edition of Olivier's Œuvres choisies (Lausanne, 1879).