The house was originally built in the late 18th century. It was later re-designed in the mid 1820s by the architect P. F. Robinson. The alterations were commissioned by industrialist John Henry Vivian. The building became part of Swansea University in 1920 and was originally the core premises of the University.