Rabbi Kahaneman was born in Kuhl, Lithuania, a small town of about 500 of which about a third were Jews. At the age of 14 he went to study Talmud at the Telshe yeshiva, where he studied Torah until he was twenty, under the direct inspiration of Rabbi Eliezer Gordon, who saw his potential. Another mentor of his in Telshe at the time was Rabbi Shimon Shkop. He then spent a half year in Novardok yeshiva, after which he spent three years in Raduń Yeshiva studying under the tutelage of the Chofetz Chaim and Rabbi Naftoli Trop. He married the daughter of the rabbi of Vidzh, and became rabbi there at the end of 1911, when his father-in-law became the rabbi of Vilkomir (Ukmergė).
With the passing of Rabbi Itzele Rabinowitz in 1919, Rabbi Yosef Shlomo Kahaneman was appointed the new rabbi of Ponevezh (Panevėžys), one of the largest centres of Jewish life in Lithuania. There, he built three yeshivas as well as a school and an orphanage. He was elected to the Lithuanian parliament. All of his institutions were destroyed and many of his students and family were killed during World War II.
Rabbi Kahaneman immigrated to the British Mandate of Palestine in 1940 and built Kiryat Ha-Yeshiva ("Town of the Yeshiva") in Bnei Brak and Batei Avot orphanages. Rabbi Kahaneman travelled widely in the diaspora to secure financial support for his yeshiva, which he constantly improved and extended. In the face of scepticism and opposition, he succeeded in turning the re-established Ponovezh yeshiva into one of the largest in the world.
He sought to take care of many orphans and tried to rescue them from the clutches of secular Zionist organizations, especially the Yaldei Tehran ("Children of Tehran") - children who escaped from Nazi Europe by walking across Europe to Tehran (including the famous Biala Rebbe - Rabbi Ben Zion Rabinowitz).