Learning music by ear is done by repeatedly listening to other musicians and then attempting to recreate what one hears. This is how people learn music in any musical tradition in which there is no complete musical notation. Many people in cultures which do have notation still learn by ear, and ear training, often through a musicianship course at a music conservatory or college, is common practice among those who use notation extensively.
Audiation involves hearing sounds mentally, although on a different level to just "hearing a song in your head". In addition to mentally hearing rhythms and pitches the skill of reproducing those sounds involves melody, harmony (chords) and bass line. In the West, learning by ear is associated with traditional and folk music, but many classical music forms throughout the world lack notation, and have therefore been passed from generation to generation by ear.
The Suzuki method of teaching music has a highly developed focus on playing by ear from a very young age. In his book "Teaching from the Balance Point," Edward Kreitman, a US based Suzuki Teacher, clearly distinguishes "learning by ear" as a separate, completely different process from "learning by rote".