Alger's parents were neighbors of the Smith's, and Alger lived with Smith and his wife, Emma. Chauncey and Ann Eliza Webb, ex-Mormons, later recalled that rumors had been whispered while Alger lived with the Smiths about Smith and Alger. Alger stopped living with the Smiths as a result of a fallout with Emma and was dismissed as their housekeeper.
The first contemporary reference to the alleged relationship was in a letter dated January 21, 1838. Oliver Cowdery wrote to his brother Warren stating that Smith had inappropriately spent time alone with Alger, referring to it as a "dirty, nasty, filthy affair." During this time Cowdery was estranged from Smith and they were disagreeing over leadership issues in the new movement.
Historian Lawrence Foster has dismissed the claim that Alger was ever married to Joseph Smith, stating that it is "debatable supposition, not an established fact". Fawn Brodie, in her famous work No Man Knows My History, also made the claim that Alger had been an affair of Smith's.
In 1903, Benjamin F. Johnson, a patriarch in the LDS church in Utah, wrote a letter to George S. Gibbs. Although Johnson was a teenager at the time, and not an intimate of Smith, he repeated the rumors he had heard about the relationship and alleged that he had been "told by Warren Parish, that he himself and Oliver Cowdery did know that Joseph had Fannie Alger as a wife Johnson also claimed that, although Alger did not join the Saints in Utah, "she did not turn from the Church nor from her friendship for the Prophet while she lived"(sic).
According to George D. Smith, Alger's marriage to Smith may have been attested by several contemporaries at the time, including Emma Smith, Warren Parish, Oliver Cowdery, and Heber C. Kimball, even though publicly the leadership of the church, including Joseph Smith and Emma denied that Joseph Smith had been a polygamist throughout their lives.