A small, private ferry dock, operated by the Gulf Islands Ferry Company already existed at Swartz Bay, when the choice was made to locate the B.C. Ferry Corporation terminal there. As a result of the decision, a paved extension of the Patricia Bay Highway to reach the terminal was undertaken.
The first scheduled sailing from the terminal, described as a "mudhole" at the time, due to the continuing construction of it, occurred on June 15, 1960.
On September 1, 1961 the company that built and used the smaller original dock at Swartz Bay was bought out by B.C. Ferries.
A fire, caused by a cutting torch, led to $500,000 worth of damage to the terminal in December, 1981. It destroyed part of the passenger walkway and a wingwall, prior to being contained by terminal staff, firefighters and the approaching Queen of Nanaimo, which utilized its hoses to help keep the blaze at bay.
Residents have expressed noise concerns over the years, and the matter has been raised in the Legislative Assembly. Concerns ranging from terminal expansion to the shooting of pigeons at Swartz Bay by ferry staff, have also been expressed.
On September 14, 2000 the ferry Spirit of Vancouver Island struck a pleasure craft just off Swartz Bay, killing two.
The terminal includes an administration building, and various kiosks run by local artisans and other vendors mainly in the warmer months. There is also a cafeteria and coffee shop in the main building.
The terminal is served by public transit through the Victoria Regional Transit System's 70 (Swartz Bay/Downtown/Airport) route. Short-and-long-term pay parking is run by a private operator under contract to BCFS. The terminal is located about 10 minutes from Victoria International Airport via Highway 17.