In 85 BC, Huo was created the Marquess of Bowang (博望侯).
In the same year, Jin, a moderating influence in the coregency, died. After Jin's death, Shangguan became increasingly jealous of Huo's powers, even though the two had been great friends, and Huo had given his daughter in marriage to Shangguan's son Shangguan An (上官安). In 84 BC, as a ploy to further strengthen his powers, Shangguan Jie gave his granddaughter (also Huo's granddaughter), then age five, in marriage to the emperor, then age 11, and she was made empress in 83 BC.
In 80 BC, the dormant conflict between Huo and Shangguan came to a head. Shangguan formed a conspiracy with Liu Dan the Prince of Yan, the Princess Eyi (鄂邑公主) (who, as the emperor's sister, had served as his caretaker), and another important official Sang Hongyang (桑弘羊) to make false allegations of treason against Huo. However, Emperor Zhao, who trusted Huo, did not act on the allegations. The conspirators then planned a coup d'etat, but were discovered. Most of the conspirators, including Shangguan, were executed, and Liu Dan and the Princess of Eyi were forced to commit suicide.
In response, Huo decided to depose the new emperor -- then an unprecedented action in Chinese history. Under an edict issued by Empress Dowager Shangguan -- Huo's granddaughter -- Prince He was deposed after just 28 days as emperor and exiled to his old principality of Changyi, but without a princely title.
There was no imperial heir in sight who fit Huo's standard of a diligent and skilled emperor. At the suggestion of another senior official Bing Ji (丙吉), Huo made a great-grandson of Emperor Wu (whose grandfather Liu Ju had been Emperor Wu's crown prince by Empress Wei but who subsequently fell out of favor and was killed in a disturbance, with his issue being removed from the imperial household), Liu Bingyi (later Liu Xun 劉詢) emperor. 27 days after Prince He was deposed, Liu Bingyi, who then was a commoner with no titles, became emperor (later known as Emperor Xuan).
In 71 BC, Huo Guang's wife Xian (顯), in order to make her daughter Huo Chengjun (霍成君) empress, poisoned Emperor Xuan's wife Empress Xu Pingjun by bribing her doctor. In 70 BC, Huo Chengjun was created empress.
In 67 BC, Emperor Xuan made his son Liu Shi (劉奭, later Emperor Yuan), by the deceased Empress Xu, crown prince, an act that greatly angered Lady Xian, who instructed her daughter to murder the crown prince. Allegedly, Empress Huo did make multiple attempts to do so, but failed each time. Around this time, the emperor also heard rumors that the Huos had murdered Empress Xu, which led him to further strip the Huos of actual power.
In 66 BC, Lady Xian revealed to her son and grandnephews that she had, indeed, murdered Empress Xu. In fear of what the emperor might do if he had actual proof, Lady Xian, her son, her grandnephews, and her sons-in-law formed a conspiracy to depose the emperor. The conspiracy was discovered, and the entire Huo clan was executed by Emperor Xuan -- an act that later drew heavy criticism from historians for its ungratefulness to Huo Guang. (e.g., Sima Guang in his Zizhi Tongjian.) (For the time being, Empress Huo was deposed but not executed, but 12 years later she was exiled; in response, she committed suicide.)
Despite the destruction of the Huo clan, Emperor Xuan continued to honor Huo Guang posthumously. In 51 BC, when he painted the portrait of 11 great statesmen of his administration in the great hall of his palace, Huo, alone among the 11, was referred to by title and family name only, which was considered an even greater honor than the honor given to the other 10.
Many later conspirators in Chinese history would often claim that they were acting in the empire's best interest, like Huo, even though few actually did. (One example -- perhaps the only one -- was the conspiracy against Emperor Shao of Southern Song Dynasty.) Conversely, when emperors wanted to accuse (and execute) officials of treason, they often euphemistically refer to them as "wanting to act like Huo Guang." Effectively, Huo set a standard of decisiveness and strength that was rarely matched and even more rarely used for the benefit of the state.