Although there are records of one Adam le Stut settled here in the 13th century, the present house, built for the Gyde family of local weavers, dates only from 1743, the design being attributed to William Halfpenny. In the late 18th century, the estate was acquired by the Lloyd Baker family, whose principal estate was established at Hardwicke Court, near Gloucester. It was subsequently occupied as a secondary house by members of the family, including Colonel Browne. Stouts Hill was the birthplace reputedly of the Gloucestershire historian, Samuel Rudder, and of the distinguished Persian scholar Edward Granville Browne.
From 1935 until 1979 it was let for use as a private boys' preparatory school in which some 100 pupils were boarders. Notable former pupils of the school include Stephen Fry, who makes a detailed reference to the school in his autobiography, Moab Is My Washpot, and Captain Mark Phillips, former husband of Anne, the Princess Royal.