A soldier in the Red Army during World War II and a designer in a shipyard, Romanov joined the Communist Party of the Soviet Union in 1944. He fulfilled several important posts in the party committee of the enterprise he was working at and later in the Leningrad city and regional party committees. In September 1970 he was elected First Secretary of the Communist Party Committee of the Leningrad Region. In this position he gained a reputation of being a good organizer and well versed in economic matters. He was elected a member of the Central Committee at the XXIVth congress of the CPSU in 1971, became a candidate member of the Politburo in 1973 and a full member in 1976.
In 1983 Romanov attracted the attention of the new General Secretary Yuri Andropov who subsequently brought him to Moscow and helped promote him in June 1983 to the highly prestigious and influential post of a secretary of the Central Committee of the CPSU responsible for industry and the military-industrial complex. During the few remaining months of Andropov's life Romanov was widely seen as one of Andropov's closest collaborators and was an ardent supporter of Andropov's comprehensive program for the reform, renewal and further development of socialism in the Soviet Union and beyond, a fact which stands in sharp contrast to the picture Gorbachev and his associates were later to paint of Romanov as a means of gaining advantage in the power struggles following Andropov's death in February 1984.
During Konstantin Chernenko's short time in office as General Secretary in 1984 – 1985, Romanov already occupied a position clearly inferior to Gorbachev, who had been styled Second Secretary of the Central Committee since February 1984 and acted as chairman of the Politburo, Secretariat and Central Committee in the course of Chernenko's long periods of absence due to his illness.
Romanov was the second youngest member of the Politburo after Gorbachev so Romanov and Gorbachev were commonly regarded to be chief rivals in the succession struggle following the death of Konstantin Chernenko in March 1985. Romanov was unable to defeat Gorbachev for the position and this ended his political career. The main reason for this outcome lies in the fact that contrary to Gorbachev who knew very well to establish excellent relationships to both the so-called "old guard" (for instance, Gorbachev enjoyed the explicit trust and support of Chernenko) as well as the young "reformers" and who was able to skillfully befriend with the various factions inside the central party leadership, Romanov for a number of reasons by far didn't dispose of the influential connections and ties of friendship that his opponent had at hand. He therefore remained in a rather lonesome outsider position in the Politburo after Andropov's death. In addition various rumors were spread at this time, most probably launched by Gorbachev and his associates, concerning Romanov's health, his character and behaviour and it was managed to unduly portray him as an opponent of reforms, a claim definitely proven to be wrong. All these factors contributed to Gorbachev's final success over Romanov and other rivals and secured his election to the post of General Secretary at the Central Committee meeting on 11 March 1985. In some other accounts (of more doubtful credibility) it has also been alleged that Romanov and two of his supporters in the Politburo were out of town and not informed of the emergency meeting held to elect Chernenko's successor and that had they been present Romanov would have had enough votes to either become the new General Secretary or block Gorbachev from attaining the position.