The Club was founded in 1906 and is located in a Georgian-style building set within its own elegant grounds, close to the town centre. It overlooks the finish line of the Henley Royal Regatta and is also opposite Leander Club.
The original building on this site dates from 1301 and was known as Fillets Court. It was the Manor of Henley-on-Thames.
Queen Anne (the consort of James I) visited in 1604. In 1643, Oliver Cromwell built a wall that still edges the garden near the river. Later that century, William of Orange, on his way to London, held his first court here. In 1998, Queen Elizabeth II left the Club by boat paid to open the River and Rowing Museum further up the river.
Phyllis Court has been completely rebuilt twice. The present building dates from 1837. The Club was partially rebuilt after a fire in 1976.