Born in Ivangorod, Shakhmatov was brought up by his uncle near Saratov. He went to a public school in Moscow and developed interest for Old Russian language and literature at an early age. At the age of 16, his articles started to appear in the most authoritative journal of Slavic studies of that time - Archiv für slavische Philologie.
Shakhmatov furthered his education at the Moscow University (1883-1887), later delivering lectures in the same institution. His first monograph, published in 1886, examined the language of ancient Novgorod charters. In 1891 he became so enthusiastic about zemstvo that he gave up his scholarly pursuits for three years and held a minor administrative office in his native village.
In 1894 Shakhmatov returned to Moscow and won great acclaim for his Ph.D. dissertation, entitled Studies in the Sphere of Russian Phonetics. Five years later, he was admitted to the Russian Academy of Sciences, and over the following years became one of the most reputable academicians. He revived the Academy's linguistic periodicals, edited the academic dictionary of Russian language and was elected to represent the Academy at the State Council of Imperial Russia and Imperial State Duma.
In 1909, Shakhmatov moved to work in the St Petersburg University as a professor. By that time, he had been elected doctor honoris causa by the Charles University, Berlin University, Polish Academy of Sciences, and many other scholarly societies.
Shakhmatov is best remembered for having pioneered textological research of early Russian chronicles, notably the Primary Chronicle. He established with a great degree of precision the stages of evolution of that key document, even attempting to reconstruct the postulated proto-version of Nestor's chronicle. His research proved seminal for subsequent generations of historians.
Shakhmatov was also responsible for publication and pioneering studies of minor or derelict Slavic languages. His studies of Slavic etymology revolved around the idea of close contacts and influences between the ancient Slavs and Celts, a hypothesis that was subsequently discarded.