Costa learned to play accordion at age 7 and was reading music three years later. Frank Oliver, Costa's high school music teacher, urged him to learn the piano after discovering that Costa had perfect pitch. Costa graduated from Carnegie Mellon University with two degrees, in music and in education. In case he failed as a musician, Costa prepared himself to teach. Following college graduation, he began work the same day as the house pianist for a radio station in Pittsburgh. Eventually he performed the same role for KDKA-TV in Pittsburgh. He provided piano and organ music for many programs, eventually teaming with Fred Rogers to arrange and perform the music heard on Mister Rogers' Neighborhood for which he served as musical director until his death in 1996.
Costa's first recording was The Amazing Johnny Costa, a Savoy LP released in 1955 and reissued on CD as Neighborhood in 1989. Although his increasingly lucrative career was beginning to bring him international attention, the amount of time away from his family and friends led him to live and perform only in western Pennsylvania. He stopped traveling and gave up his job as musical director of The Mike Douglas Show. He returned to Pittsburgh and remained there for the rest of his life.
Costa served as musical director, arranger, and pianist for the children's television program Mister Rogers' Neighborhood from the program's debut in 1968 until his death in 1996. The program's creator and host, Fred Rogers, regarded Costa as one of the most gifted musicians he had ever met. Costa was one of the world's most talented jazz pianists. Rogers' choice was surprising because as Costa's style was regarded as too complicated and sophisticated for a children's program. But Costa insisted on not playing "baby" music.
Each day, Costa and his trio played live in the studio for the filming. In addition to the show's recognizable main theme, they played the trolley whistle, Mr. McFeely's frenetic Speedy Delivery piano plonks, the vibraphone flute-toots as Fred fed his fish, dreamy celesta lines, and Rogers' entrance and exit tunes.
Costa died of anemia (not leukemia as often misreported) at age 74 .
After his death, much of the music heard on the program continued to be Costa's. The show's closing credits also continued to list Costa as its Musical Director.