When he was young, Passer was very shy, and had a dream of becoming an actor, but consented to his father's wishes by attending J. Lauritzen's sea training school near Svendborg in 1944. But since he had ongoing problems with seasickness, he later started at De frederiksbergske teatres Elevskole.
During the 1950s he formed a duo with his colleague and friend Kjeld Petersen. Their revue sketches, based upon the contrast between Petersen’s mixture of joviality and desperate anger and Passer’s deadpan responses, are still classics. The sudden death of Kjeld Petersen in 1962 led Passer to avoid revues for some years, but he built up an individual reputation and in 1967 he returned to the revue gaining new victories. Many thin jokes in the scripts were greatly improved by his performance. In particular, his many amiable eccentrics and “nature experts” together with his sketch roles as a baby and as a nonsense "Russian"-speaking clown made him famous. From his later years must be mentioned an almost silent sketch in which he portrays a man’s vain attempt to stop smoking (also shown in West German TV). It was told that he could speak any language, however he wouldn't understand any of it, which was one of his good qualities.
He was often referred to as a loud actor in spite of the fact that under-acting was responsible for much of his force. A Danish critic, Jens Kistrup, once said that one of the secrets behind the comedy of Passer was its combination of elements that are normally regarded as incompatible. He possessed noisiness and discretion, loudness and quietness, boundlessness and complete control, craziness and softness — all this combined with a special intimacy with the audience. Among his inspirations he mentioned Joe E. Brown but he was also known for his admiration of the British comedian Tommy Cooper. In his films, which were of very mixed quality, he often played kind and somewhat crazy "Everymen" or antiheroes. Among his best movie roles were stage roles transferred to film; here must be mentioned the hero in Charley’s Aunt (1959), Celestin-Floridor in Frøken Nitouche (1963) and Leopold in Summer in Tyrol (i.e. The White Horse Inn, 1964).
Numerous Danish actors see him as a role model. In day-to-day life he was quite shy in behaviour, somewhat the opposite of his theatrical appearances. He died during a sketch in which he was portraying the Danish rock singer Kim Larsen.