He studied music at the Athens Conservatoire as well as in Brussels and Berlin, with Ferruccio Busoni among his teachers. From 1921 to 1925 he assisted Erich Kleiber at the Berlin State Opera and then took a number of posts in Greece. At a 1930 concert with the Berlin Philharmonic, he played the solo part of Sergei Prokofiev's Piano Concerto No. 3 and conducted the orchestra from the keyboard, becoming one of the first modern musicians to do so.
Mitropoulos made his U.S. debut in 1936 with the Boston Symphony Orchestra, and he later settled in the country, becoming a naturalized citizen of the United States in 1946. From 1937 to 1949, he served as the principal conductor of the Minneapolis Symphony Orchestra.
In 1949 Mitropoulos began his association with the New York Philharmonic, the peak of his orchestral career. He was initially co-conductor with Leopold Stokowski, and became the sole music director in 1951. Mitropoulos recorded extensively with the Philharmonic for Columbia Records and sought to reach new audiences through appearances on television and conducting a week of performances at the Roxy Theatre, a popular movie theatre in New York. Mitropoulos expanded the Philharmonic's repertoire, commissioning works by new composers and championing the symphonies of Gustav Mahler. In 1957 he was succeeded as the Philharmonic's conductor by his former protégé, Leonard Bernstein.
In addition to his orchestral career, Mitropoulos was an equally important force in the operatic repertoire. He conducted opera extensively in Italy and from 1954 until his death in 1960 was the principal conductor of the Metropolitan Opera in New York, although the Met did not officially use that title at the time. His musically incisive and dramatically vivid performances of Puccini, Verdi, Richard Strauss and others remain models of the opera conductor's art. The Met's extensive archive of recorded broadcasts preserves many of these fine performances.
Mitropoulos's series of recordings for Columbia Records with the New York Philharmonic included a rare complete performance of Alban Berg's Wozzeck. Many of these have been reissued by Sony Classics on CD, including most recently his stereo recordings of excerpts from Prokofiev's Romeo and Juliet. For RCA he recorded with the Minneapolis Symphony during the 78-rpm era. He was also represented on the Cetra label, most notably with an early recording of Richard Strauss's Elektra.
He was noted for having a photographic memory (which enabled him to conduct without a score, even during rehearsals) and for his monk-like life style due to his deeply religious beliefs (Greek Orthodox).
Mitropoulos never married. He was "quietly known to be homosexual" and "felt no need for a cosmetic marriage". Among his relationships reportedly was one with Leonard Bernstein.
He died in Milan, Italy at the age of 64, while rehearsing Gustav Mahler's 3rd Symphony. One of his very last recorded performances was Verdi's La Forza del Destino with Giuseppe di Stefano, Antonietta Stella and Ettore Bastianini at Vienna on 23 September 1960. A recording exists of the performance of Mahler's Third Symphony given by Mitropoulos with the Cologne Radio Symphony on October 31, 1960, just two days before his death.
His compositions include a piano sonata and other works.