Colwood's best known landmark is Hatley Castle, a castle-like building originally built as a residence for James Dunsmuir, premier and later Lieutenant Governor of British Columbia. Colwood almost became the seat of the Commonwealth Crown in 1940, when, after the eruption of World War II, it had originally been planned that King George VI, Queen Elizabeth, and their children, Princesses Elizabeth and Margaret, would leave London, UK, for Colwood, to reside at Hatley Castle, which the federal government had purchased for use as a Royal Palace for the King and his family. However, it was decided that morale in Britain would be diminished if the Royal Family were to abandon the European front, and they remained in London and Windsor. Following the collapse of those plans, the site was turned over to the Canadian Armed Forces, and from 1941 until the mid 1990s operated as Royal Roads Military College. The military college was closed in the 1990s and the lands have since formed the campus for Royal Roads University which opened in 1995. The original residence is surrounded by extensive and attractive gardens, including a Japanese garden.
The city is the site for many regional amenities, including the Juan de Fuca Recreation Centre, Bear Mountain Arena, and the library facilities for the Western Communities.
As it lies geographically to the south of Victoria, Colwood is Western Canada's southernmost city.