Swarthmoor Hall was built by a lawyer named George Fell about 1568. His son Thomas inherited it. In 1634 Margaret Askew married Thomas, later named Vice Chancellor of the Duchy of Lancaster and an influential supporter of parliament, and moved into the Hall.
George Fox visited the Hall in 1652. Thomas Fell was travelling as a judge, but Fox had an audience with Margaret Fell, who became interested in his new doctrines. She arranged for him to preach in St. Mary's Church in nearby Ulverston and at the Hall. During his time there, many people were convinced of the truth of his teachings.
When Thomas Fell returned home, he was persuaded by his wife and some others to listen to Fox, who successfully appealed to his pro-parliamentary sentiments. Fell was never totally convinced by Fox's religious teachings, but he did allow his home to be used as a meeting house for the early Friends. They did so until 1691 when a meeting house was built nearby.
Thomas Fell died in 1658. Eleven years later George Fox married the widowed Margaret Fell and, when not traveling, occasionally lived at Swarthmoor. Fox died in London in 1691, but Margaret died at the Hall in 1702. It now belongs to the Religious Society of Friends.