Empress Wang was sonless, and Liu suggested to her that she should request that Emperor Gaozong's oldest son, Li Zhong, whose mother Consort Liu was of low birth, be created crown prince, so that Li Zhong would be grateful to her. Empress Wang did so, and Emperor Gaozong agreed, after Liu Shi lobbied his fellow chancellors, including Emperor Gaozong's powerful uncle Zhangsun Wuji. Later in 652, Emperor Gaozong created Li Zhong crown prince.
By 654, however, Emperor Gaozong's favors for Empress Wang had waned, particularly in light of his increasing favor for his concubine Consort Wu. The situation was exacerbated by the fact that Empress Wang was not good at earning the loyalty of the other concubines and the ladies in waiting, while Consort Wu was able to, and also that Liu Shi and his sister and Empress Wang's mother Lady Liu were not respectful to the other concubines. In 654, in fear, Liu requested to be relieved of his chancellor post, and Emperor Gaozong agreed, making him the minister of civil service, no longer a chancellor.
In 655, Consort Wu falsely accused Empress Wang and Lady Liu of using witchcraft to try to regain favor for Empress Wang. In response, Emperor Gaozong barred Lady Liu from the palace and demoted Liu out of the capital Chang'an to be the prefect of Sui Prefecture (遂州, roughly modern Suining, Sichuan). As he travelled through Qi Prefecture (岐州, roughly modern Baoji, Shaanxi), the prefectural secretary general Yu Chengsu (于承素) submitted a report accusing Liu of revealing palace secrets, and Emperor Gaozong further demoted Liu to be the prefect of Rong Prefecture (榮州, roughly modern Zigong, Sichuan), a smaller prefecture. Later that year, Emperor Gaozong deposed Empress Wang and replaced her with Consort Wu, and the former Empress Wang was soon executed along with her ally Consort Xiao by order of the new Empress Wu.
In 657, Empress Wu's allies Xu Jingzong and Li Yifu further accused the chancellors Han Yuan and Lai Ji, both of whom had opposed Empress Wang's removal and Empress Wu's ascension, of conspiring with the former chancellor Chu Suiliang, who was even more vocal in his opposition and who had been demoted as a result. Han, Lai, and Chu were all demoted to be prefects of distant prefectures, and Liu was also further demoted to be the prefect of the extremely distant Xiang Prefecture (象州, roughly modern Laibin, Guangxi).
In 659, Empress Wu carried out even more reprisals -- this time, having Xu falsely accuse Zhangsun, who had shown implicit disapproval of her ascension, of treason, and Chu (who had died at this point), Liu, and Han of encouraging Zhangsun. Zhangsun was exiled (and soon forced to commit suicide), while Liu and Han were demoted to commoner rank. In fall 659, Emperor Gaozong further sent imperial messengers to arrest Liu and Han, as well as Zhangsun Wuji's cousin Zhangsun En (長孫恩), but even while imperial messengers were on the way changed the order to have Liu and Han executed instead. Liu's clan members were exiled to the modern Guangdong and Guangxi region, while his immediately family was specifically confiscated to become servants at Gui Prefecture (桂州, roughly modern Guilin, Guangxi).