Ashbee was born in Southwark, London, and was married in Hamburg, Germany in 1862. He was by occupation a textile trader. He travelled extensively during his life, including Europe, Japan, and San Francisco, collaborating with Alexander Graham on Travels in Tunisia, published in 1887. He was an avid book collector, with perhaps the world's most extensive collections of Cervantes and erotica.
Ashbee was a part of a loose intellectual fraternity of English gentlemen who discussed sexual matters with a freedom that was at odds with Victorian mores; this fraternity included Richard Francis Burton, Richard Monckton Milnes, Algernon Swinburne, and others. He also amassed thousands of volumes of pornography in several languages. He wrote on sex under the pseudonyms "Fraxinus" (Ash) and "Apis" (Bee), and sometimes combined them as "Pisanus Fraxi".
Ashbee's will left his entire collection to the British Museum, with the condition that the erotic works had to be accepted along with the conventional items. Because the trustees wanted the materials related to Cervantes, they decided to accept the bequest. The trustees exploited a loophole to destroy some of the erotica, although some of the works are in the British Library, including a work by William Simpson Potter.
His family life grew unhappier as he aged. As he became more conservative, his family moved with the times and became more progressive. "The 'excessive education' of his daughters irritated him, his Jewish wife's pro-suffragism infuriated him, and he became tragically estranged from his socialist, homosexual son, Charles" (the designer Charles Robert Ashbee).
Ashbee's most famous works were his three bibliographies of erotic works:
A character based on him is central to Sarah Waters's award-winning novel Fingersmith: a man obsessively collecting and indexing pornography and works about human sexuality, in an atmosphere of oppressive Victorian hypocrisy.