In 1610, Ryonan-zenji founded it at Akasaka and in 1636 it was relocated to its present location. The temple was located directly on the Tokaido Road to the sea, and for that reason it was named Kaijō Zenrin (literally, "the Zen forest above the sea"). The words were embazoned prominently on the Sanmon or main gate which long ago disappeared. In the Edo period, Tōzen-ji was considered the family temple of various clans, including the Date clan of Sendai, the Ikeda clan of Omi province, the Inaba clan of Usuki Domain in Bungo province, the Suwo of Shinshu, the Tamura of Ichinoseki, and the Mori clan of Saeki in Bungo.
In 1859, the first British legation in Japan was built in the precincts of the temple. Even now, the sword cuts and bullet marks in the attack of the samurai of the Mito han remain in the pillar of the Okushoin (the drawing room in the back) and the genkan. Rutherford Alcock, the first British Minister to Japan, recorded his impressions of Tōzen-ji in his book, The Capital of Tycoon (1863).