She was the fourth child and second daughter of the Lady Barbara Palmer, who was by then separated but still married to Roger Palmer, 1st Earl of Castlemaine. Castlemaine likely did not father any of his wife's children; rather, Charlotte and her siblings were probably the offspring of their mother's royal lover, Charles II. The king acknowledged her as his daughter and so she bore the surname of Fitzroy - "son of the King".
She was the favorite niece of James II, brother of Charles II. "We know but little of her except that she was beautiful; she "rivaled her mother in beauty, but was far unlike her in every other respect. As the Duchess of Cleveland was known for her diabolical nature, the implication here is that Charlotte was a sweet-tempered and pleasing person; one memoirist attests to that assumption, describing Lady Lichfield as "a very good and virtuous lady." It is said that the king had a greater value and love for this lady than he had for his other children.
Lady Charlotte was contracted to Sir Edward Lee shortly before her tenth birthday and married at thirteen years of age. When Charles Stewart, 6th Duke of Lennox died in 1673, Sir Edward was created Earl of Lichfield.
They had by some accounts thirteen sons and five daughters, and by other accounts nine sons and four daughters, including: