In 1865 he resigned elected office to become Secretary of the Interior under President Andrew Johnson, an appointment he held until 1866. As secretary he announced that he intended to "clean house" and fired "a considerable number of incumbents who were seldom at their respective desks". Amongst this group was the poet Walt Whitman, then working as a clerk in the department, who received his dismissal note on June 30, 1865. Harlan had found a copy of Leaves of Grass on Whitman's desk as the poet was making revisions and found it to be morally offensive. "I will not have the author of that book in this Department", he said. "If the President of the United States should order his reinstatement, I would resign sooner than I would put him back. 29 years later, however, he defended his actions, saying that Whitman was dismissed solely "on the grounds that his services were not needed".
Harlan resigned from the post in 1866 when he no longer supported the policies of President Johnson. He was elected again to the United States Senate in 1867 and served until 1873.
From 1853 to 1855, Harlan was president of Iowa Wesleyan College in Mount Pleasant, Iowa, where, following his career of public service, he resided until his death in 1899. Along with pioneer Iowa governor Samuel Kirkwood, Harlan's sculptured likeness is maintained among the two coveted statues apportioned to each state on display under the rotunda in Statuary Hall in the United States Capitol in Washington, D.C.