A daughter of Viscount Dillon, in 1826. Her ancestors had had pronounced Jacobite leanings. Born in Halifax, Nova Scotia she was a great-great-great-great-granddaughter of King Charles II by his mistress Barbara Villiers. Her grandmother, née Henrietta-Maria Phipps, was also a descendant of James II of England by Catherine Sedley. Another ancestor was Maréchal de camp Count Arthur Dillon (1670–1733), a supporter of the "Old Pretender".

Before her marriage, she had lived in Florence and had attended the receptions of the Countess of Albany, the widow of Charles Edward, the Young Pretender. In London, she had great influence in social and political circles. When he was Patronage Secretary, her husband was described by Lord Palmerston as joint-whip with Mrs Stanley. Later in life, Lady Stanley of Alderley helped to found the Women's Liberal Unionist Association; and she was a strenuous worker for the higher education of women, helping to establish Girton College, Cambridge, the Girls' Public Day School Trust, and the Medical College for Women.