The organisation was founded in the 1850's during the upheavals of the Taiping Rebellion. It was one of a number of rebel groups to arise during this period, either affiliated with or proclaiming support for the Taiping administration. The name ("Small Swords") refers to daggers used by warriors or martial artists in close combat. It is believed to be linked to triads. The society consisted mainly of natives from Guangdong and Fujian, including Li Shaoqing, Li Xianyun and Pan Yiguo, directors of some of the huiguan or native place associations of Shanghai.
In 1853, the Society occupied the walled (Chinese) city of Shanghai, and occupied most of the Chinese sections of the city. They did not, however, invade the foreign concessions. Large numbers of Chinese refugees from surrounding areas flooded into the foreign concessions in this period, dramatically increasing the population there, and giving rise to the prevalent "Nongtang" or "Shikumen"-style housing which came to dominate Shanghai by the early 20th Century.
The Society's headquarters were in the Yuyuan Gardens of Shanghai, at the heart of the old city and today a popular tourist attraction and shopping district. There is a small museum displaying artefacts of the Society in the gardens.
Conflict broke out between the Fujian and Guangdong factions, over whether they should leave with the loot they had acquired. At first, the British and American authorities remained neutral, but the French supported the imperial government. However, some British and American sailors joined up with the Small Swords Society. When French troops were sent in to support Qing imperial troops, this caused the embarrassing situation of whites fighting whites (which was not done among (transplanted) Europeans). The British and American authorities then declared the sailors' actions as illegal, and joined in support for the imperial armies.