It was Freskin who built the great earthwork and timber motte-and-bailey castle in c.1140. The motte was an immense man-made mound with steeply sloping sides and a wide and deep ditch that surrounded the base. Timber buildings would have stood on its flat top and would have been further protected by a wooden palisade placed around the edge of the summit. The motte was accessed from the bailey. This is a wide stretch of earth elevated above the surrounding area but not as high as the motte. At Duffus, the motte would have been reached by steps set into the mound. The motte contained the buildings necessary to sustain its inhabitants - brew and bake houses, workshops and stables - as well as the living accommodation.
In 1305, it was recorded that Reginald Cheyne received a grant from King Edward I of England of 200 oaks from the royal forests of Longmorn and Darnaway 'to build his manor of Dufhous' demonstrating that a large construction project was being carried out. The wood would have been needed for scaffolding, flooring and roofing of the new stone fortress. It is known that the original castle was burned down by Andrew Moray in the summer 1297 who descended on it with what was described as in a letter to King Edward I of England as “a very large body of rogues”, because it held a garrison of King Edward’s English troops and this had been the impetus for building a more secure castle of stone.
A two-storey rectangular tower was built on the motte and was the main residence. The first floor held the lord’s hall, with a latrine and bed chambers. The ground floor was the main storage space and also accommodated the lord’s household. The tower was built as a defensive structure with a small number of narrow windows. There was only the one entrance on the ground floor which also housed a portcullis. On the second floor, two doors exited onto the walkway of the curtain wall. This wall completely enclosed the bailey. The put-log holes built into the curtain wall indicate the presence of a number of buildings. On the north side a later building was erected that housed a kitchen, a great hall with reception room and the great chamber bedroom. It is possible that this building was constructed by the Sutherlands. It is not known when the serious subsidence took place but evidence of repairs to the tower are evident before it slid down the motte. The tower shows no further repairs and may have collapsed early on but the newer hall became the main residence. This building shows continued alterations over time. In 1689, John Graham, 1st Viscount of Dundee was a guest of Lord Duffus just before the battle of Killiekrankie and would be one of the last important visitors before the castle’s abandonment.