Castle Cary was the site of a 12th century castle that was built on the hill behind the town. This was erected during the reign of King Stephen, but has now completely disappeared apart from a few earthworks. The town grew around the mediaeval weaving industry and is home to a horsehair weaving factory.
The town is on the Monarch's Way long distance footpath.
The largest church in the town is All Saints', which date from 1470 and is notable for its high steeple, which contains six bells dating from 1760 and made by Thomas Bilbie of the Bilbie family. It has been designated by English Heritage as a grade II* listed building. There is also a Methodist church and St Andrew's in the neighbouring town of Ansford.
Notable people from the town include the 18th century diarist James Woodforde who was curate between 1765 and 1775 and Douglas Macmillan, founder of the Macmillan Cancer Relief charity. The Macmillan Way walking trail passes through the town.
The town has become increasingly popular with people moving out of London and looking for somewhere quiet to raise a family. The high speed railway line to London, good local schools and services, along with an attractive rural setting have all played their part in the recent growth of the town. Nearby is Wyvern Waste's recycling centre, based on the old army camp at Dimmer. It holds educational tours around the site for schools.
The Living History Group is an active group of local amateur historians in the town who have published several books concerning the history of Castle Cary and its personalities. In 1900, for example, Castle Cary cricket club provided five players for the Devon and Somerset Wanderers team that won the only Olympic cricket title. The Cricket Club, are to mark their 175th anniversary in 2012, with a celebration of the Castle Cary players that played in 1900, whilst the London Games take place.
On 29th May 2008 Castle Cary (and a large part of Somerset) were hit by a rapid flash flood after a violent downpour shortly after midday. The local drainage was soon overwhelmed and roads were covered in several inches of water flowing towards the centre of town. Water inundated the Horse Pond Inn, the White Hart public house and the Co-Op supermarket.
The town has a primary and a secondary school.