The original castle seems to have been a sub-rectangular/oval enclosure overlooking the River Tawe on the east, surrounded on the north, west and south sides by a larger sub-rectangular outer bailey. The inner bailey probably contained a motte but the other view is that it was a ring work. The new castle was attacked by the Welsh in 1116 but the inner castle held.
After various other unsuccessful attacks the castle fell in 1217 but was restored to the English in 1220 as part of the settlement between Llywelyn ap Iorwerth and Henry III of England. Immediately after this the inner castle was probably walled in stone with at least one tower.
Later in the 13th century the large outer bailey was also walled. The only visible remains are two sides of a rectangular "new castle" built in the South East corner of the outer bailey in the late 13th/early 14th century. The south face (which ends in a tall garderobe tower) is capped with an elegant series of arcades at the wall-head, which are similar to structures at the Bishop of Saint David's palaces at Lamphey and St David's.
By then the castle had lost its military importance. It is not known whether it fell to allies of Owain Glyndŵr early in the 15th century.
In the 18th and 19th centuries parts of the castle were variously used as a market, a town hall, a drill hall and a prison. Part of the interior of the new castle was demolished early in the 20th century in the construction of a newspaper office. The remains have now been consolidated and opened up to view from the street.