Born to Jewish parents in Moscow, where his father had just opened a small factory, he studied piano first with his mother, and later with Alexander Villoing. In the 1840s he and Anton were brought to Berlin by their mother, where they studied under Siegfried Dehn and attracted the interest and support of Mendelssohn and Meyerbeer. He was responsible for founding the Moscow Conservatory on September 1, 1866 — he was also its director. He was generally regarded as one of the greatest pianists of his time, although now his reputation is overshadowed by his brother's. Nevertheless, his pianistic style was quite at odds with that of his fiery brother. Nikolai instead opted for a restrained classicism more in line with the musical values of Clara Schumann than Franz Liszt.
While holding his Moscow post, Nikolai persuaded Tchaikovsky to write for him the celebrated Piano Concerto No. 1. According to Tchaikovsky's letters, Rubinstein was unimpressed with the work, and would only perform it if rewritten. Tchaikovsky refused, and the work was premiered instead by the pianist Hans von Bülow. Nevertheless, Tchaikovsky wrote his Piano Trio in A minor in Rubinstein's memory after he died in Paris.