He was born in Camberwell Grove, Camberwell, London, England, but lived in the United States with his mother, whose marriage had been annulled at the order of the French emperor. With the annulment came the rescission of Patterson's son's right to carry the Bonaparte name; the ruling was later reversed by Napoleon III.
There is some speculation that he is the reason that the 11th Congress of the United States in 1810 proposed an amendment to the U.S. Constitution to strip an American citizen of his or her citizenship if he or she accepts a title of nobility from a foreign nation. The amendment came close to being ratified by the legislatures of enough states to become part of the U.S. Constitution, lacking the approval of only two.
He graduated from Mount St. Mary's College (now Mount St. Mary's University) in 1817.