The track has been described by Bowie as intending to evoke the misery of the Berlin Wall, being a portrait piece like the other music on Side Two of Low. The principal melody is an adaptation of the tune "Scarborough Fair".
Bowie plays all instruments on the recording, the album's only solo track, including several percussion instruments and synthesizers. His voice is also present in a wordless chorus. Its minimalistic rhythm has been seen as bearing the influence of composer Philip Glass, who would adapt some of the music from Bowie's album for his Low Symphony in 1992.
While Brian Eno and NME editors Roy Carr and Charles Shaar Murray have suggested that "Weeping Wall" began life as part of Bowie's aborted soundtrack to The Man Who Fell to Earth (1976), the composer himself maintains that the piece was composed especially for Low.