Rabbi Berlin was born into a family of Jewish scholars renowned for its Talmudic scholarship. His father Jacob, while not a rabbi, was a Talmudic scholar; his mother was directly descended from Rabbi Meir Eisenstadt. Although initially a weak student, legend has it that Rabbi Berlin applied himself to his studies after overhearing his parents debating whether he should pursue a trade.
His first wife was the daughter of Rabbi Yitzchok of Volozhin, the son of Rabbi Chaim Volozhin. His second wife was his niece, a daughter of Rabbi Yechiel Michel Epstein, the author of the Aruch haShulchan. A son from his first marriage, Rabbi Chaim Berlin, became the rabbi of Moscow, a daughter married Rabbi Refael Shapiro, and his son from his second marriage was Rabbi Meir Berlin (later Bar-Ilan).
Rabbi Berlin led the yeshiva in Volozhin (in what is presently Belarus), then the largest institution of its kind, from 1854 to its closure in 1892. Despite the destruction (twice) of the town and the yeshiva building in large fires, its enrollment increased steadily under his leadership, and the yeshiva would produce a number of prominent rabbinic figures who led Eastern European Jewry until World War II. Amongst them was Rabbi Shimon Shkop.
In Volozhin, his leadership was contested by the popular Rabbi Joseph Dov (Yoshe Ber) Soloveitchik, whose style of Torah study differed substationally from Rabbi Berlin's. Rabbi J.D. Soloveitchik ultimately became rabbi of Slutsk, Warsaw and Brisk, where he founded the rabbinical dynasty that still carries his name.
In 1892, the Volozhin yeshiva shut down. There are several explanations and differing versions of events:
In any event, after the closure, Rabbi Berlin traveled to Vilna and other cities, trying to clear the yeshiva's debt.
In the last few months of Rabbi Berlin's life he suffered from diabetes and the consequences of a stroke. While he intended to travel to the Land of Israel, his medical condition made this impossible. He spent his last weeks in Warsaw, and is interred in a cemetery there.
Rabbi Berlin had a traditionalist approach to Torah study that was at odds with the highly analytical style of lomdus ("learned intellectual analysis") that was pioneered by Soloveitchik.
Politically, he favored Jewish settlement of the Land of Israel (Eretz Yisrael), then under the control of the Ottoman Empire; he was initially a member of the Chovevei Tzion movement (founded by his contemporary Rabbi Zvi Hirsch Kalisher), but later distanced himself from them.