The building was originally part of a much larger structure dating from approximately 1740, much of which was pulled down at the beginning of the 19th century. The house was rented for a very low rent and restored in the 1930s and 1940s by the photographer and designer Cecil Beaton, who detailed it in his book Ashcombe: The Story of a Fifteen-Year Lease.
Beaton entertained lavishly at Ashcombe House, and his houseguests included many notable people of the time, including actors and artists. Guests included Tallulah Bankhead, Diana Cooper and Lord Berners. Artists Rex Whistler, Salvador Dalí and Augustus John painted murals in the house, and Dali used it as the backdrop of one of his paintings.
Beaton's lease expired in 1945 and he was heartbroken to be forced to leave the house. Beaton's landlord, Hugh Borley, then lived in the house until the early nineties, when it was sold in a private sale. When it came up for sale in 2001, the first time it had been on the open market since World War I, there was a lot of interest. Madonna and Guy Ritchie were the successful purchasers; in May 2008 it was reported that the couple were considering selling it.
The grounds of the house are noted for their re-established wildlife, including fallow deer. The grounds are also noted as one of the top game bird shooting venues in Britain. The Field magazine voted it one of Britain's ten top venues for pheasant shooting. Public rights of way run through the grounds; these are open to the public all year round.