The son of Robert Pemberton Milnes, of Fryston Hall, Yorkshire, and the Hon. Henrietta Monckton, daughter of the fourth Lord Galway, he was born in London. He was educated privately, and entered Trinity College, Cambridge, in 1827. There he was drawn into a literary set, and became a member of the famous Apostles Club, which then included Alfred Lord Tennyson, Henry Hallam, Richard Chenevix Trench, Joseph Williams Blakesley, and others. After taking his degree, Milnes travelled abroad, spending some time at the University of Bonn. From there he went to Italy and Greece, and published in 1834 a volume of Memorials of a Tour in some Parts of Greece, describing his experiences.
He returned to London in 1837, and was elected to Parliament as member for Pontefract in the Conservative interest. His parliamentary career was marked by much strenuous activity. He interested himself particularly in the question of copyright and the conditions of reformatory schools. He left Prime Minister Sir Robert Peel's party over the Corn Law controversy, and was afterwards identified in politics with Palmerston, who made him a peer in 1863.
His literary career was industrious and cultured, without being exceptionally distinguished. Church matters had always a claim upon him: he wrote a striking tract in 1841, which was praised by Newman; and took part in the discussion about "Essays and Reviews," defending the tractarian position in One Tract More (1841). He published two volumes of verse in 1838, Memorials of Residence upon the Continent and Poems of Many Years, Poetry for the People in 1840 and Palm Leaves in 1844. He also wrote a Life and Letters of Keats in 1848, the material for which was largely provided by the poet's friend, Charles Armitage Brown.
Milnes' poetry is meditative and delicate; some of his ballads were among the most popular of their day. His chief distinctions were his keen sense of literary merit in others, and the judgment and magnanimity with which he fostered it. He was surrounded by the most brilliant men of his time, many of whom he had been the first to acclaim. His reputation rests largely on the part he played, as a man of influence in society and in moulding public opinion on literary matters, in connection with his large circle of talented friends. He secured a pension for Tennyson, helped to make Ralph Waldo Emerson known in Britain, and was one of the earliest champions of Algernon Swinburne. He helped David Gray by writing a preface for The Luggie. A less public part of his life was his apparently almost unsurpassed collection of erotic books, now in the British Library.
He was, in the traditional sense, a patron of literature, who never abused the privileges of his position. Milnes married in 1851 the Hon. Annabel Crewe (d. 1874). He died at Vichy, and was buried at Fryston. His son, Robert, was created Earl of Crewe in 1895.