Ehle enlisted in the United States Army during World War II, serving as a rifleman. Following his military service, Ehle went on to study at the University of North Carolina at Chapel Hill, receiving a bachelor of arts degree in Radio, Television, and Motion Pictures in 1949 and later a Master of Arts degree in Dramatic Arts (1953). Ehle also served on the faculty of the University of North Carolina at Chapel Hill from 1951 to 1963. During his tenure at UNC-Chapel Hill, Ehle wrote plays for the American Adventure series that played on NBC Radio and began writing his first novel.
Ehle's first novel, Move Over Mountain, was published by Hodder & Stoughton of London in 1957. The following year, Ehle returned with a biography The Survivor: The Story of Eddy Hukov. In 1964, Harper & Row published perhaps his most well known book, The Land Breakers. The book is a fictional account set in the late 18th century that traces the story of the first white pioneers to settle in the Appalachian wilderness of the mountains of Western North Carolina. The Land Breakers, out of print for several decades, was republished in 2006 by Press 53, a small imprint in Winston-Salem, North Carolina. The Land Breakers also marked the beginning of a seven part series of historical fiction about the Appalachian region.
Two of Ehle's eleven novels, The Winter People and The Journey of August King, have been adapted into film. Among his six works of non-fiction is the 1965 book The Free Men, which is a first-person chronicle of the desegregation struggle in Chapel Hill, North Carolina at the height of the civil rights movement of the 1960s.
The University of North Carolina at Chapel Hill Library, Manuscripts Department, maintains the John Ehle Papers, an archive which contains drafts, notes, correspondence, and other materials pertaining to Ehle's many books. The collection also includes a large collection of audio recordings of interviews, video, and photographs which document the civil rights activities observed by Ehle while he was writing The Free Men.
Ehle is a member of the North Carolina Literary Hall of Fame, and has received numerous awards, including: the Thomas Wolfe Prize, the Lillian Smith Book Award, and the Mayflower Award.
John Ehle's public service also reached beyond North Carolina to the national level. From 1964 to 1966, Ehle served as an advisor on President Lyndon B. Johnson's White House Group for Domestic Affairs. From 1965 to 1968 Ehle was a member of the United States National Committee for UNESCO. He also served on the National Council for Humanities (1966-1970).