The nucleus of the estate was a small farm of 100 acres (0.4 km²), called Cartleyhole, nicknamed Clarty (i.e., muddy) Hole, and was bought by Scott on the lapse of his lease (1811) of the neighbouring house of Ashestiel. He first built a small villa (now the western end of the house) and named it Abbotsford, creating the name from a ford nearby where previously abbots of Melrose Abbey used to cross the river. Scott then built additions to the house and made it into a mansion, building into the walls many sculptured stones from ruined castles and abbeys of Scotland. In it he gathered a large library, a collection of ancient furniture, arms and armour, and other relics and curiosities, especially connected with Scottish history.
The last and principal acquisition was that of Toftfield (afterwards named Huntlyburn), purchased in 1817. The new house was then begun and completed in 1824.
The general ground-plan is a parallelogram, with irregular outlines, one side overlooking the Tweed; and the style is mainly the Scottish Baronial. Into various parts of the fabric were built relics and curiosities from historical structures, such as the doorway of the old Tolbooth in Edinburgh.
Scott had only enjoyed his residence one year when (1825) he met with that reverse of fortune which involved the estate in debt. In 1830 the library and museum were presented to him as a free gift by the creditors. The property was wholly disencumbered in 1847 by Robert Cadell, the publisher, who cancelled the bond upon it in exchange for the family's share in the copyright of Sir Walter's works.
Scott's only son Walter did not live to enjoy the property, having died on his way from India in 1847. Among subsequent possessors were Scott's son-in-law, John Gibson Lockhart, J. R. Hope Scott, Q.C., and his daughter (Scott's great-granddaughter), the Hon. Mrs Maxwell Scott.
Abbotsford gave its name to the "Abbotsford Club", a successor of the Bannatyne and Maitland clubs, founded by William Barclay Turnbull in 1834 in Scott's honour, for printing and publishing historical works connected with his writings. Its publications extended from 1835 to 1864.
The house was opened to the public in 1833, but continued to be occupied by Scott's descendants until the death of Dame Jean Maxwell Scott, the great, great, great granddaughter of the writer the previous year in 2004.
Scottish Borders Council is considering an application by a property developer to build a housing estate on the opposite bank of the River Tweed from Abbotsford, to which Historic Scotland and the National Trust for Scotland object.
In keeping with its many Walter Scott references, Rose Street in Edinburgh has a bar called the "Kenilworth", along with one named the "Abbotsford".