Dolvin was born in 1912 in Richland, Georgia as the youngest of Mary Ida Nicholson (1871–1951) and James Jackson Gordy's (1863–1948) nine children. Her sister Lillian Gordy Carter was the mother of Jimmy Carter, the 39th President of the United States.
After graduating from the Georgia College for Women in Milledgeville, Georgia, Dolvin taught primary school. In 1938, she married William Jasper Dolvin and moved to Roswell, Georgia. Her husband was an elementary school principal and insurance agent, and Dolvin Elementary School in Alpharetta, Georgia is his namesake.
In 1951, Emily Dolvin participated in the organization of the Roswell Youth Recreation Committee. This committee created the Roswell Recreation and Parks department. She was also the inaugural chairperson of the Roswell Historical Society upon its establishment on October 28, 1971. The Roswell Rotary Club named her a William Watt Fellow, and she was the first female member of that organization.
In 1966, Dolvin became involved in the support of the political career of her nephew Jimmy and was often referred to as Jimmy Carter's Aunt Sissy. In 1970, she was a staff member, host, and delegate for the Georgia Democratic Party, and she was the inauguration reception chairman for Carter's inauguration as governor of Georgia in 1971, and served as the volunteer coordinator for Carter’s gubernatorial campaigns. Dolvin also served as a member of the Commission on the Status of Women from August 11, 1972 to April 1, 1974.
During Carter's 1976 presidential campaign, Dolvin campaigned throughout the United States as one of many volunteers that comprised the grass roots Peanut Brigade. However, she was not an official member of the Peanut Brigade. She was campaigning as a member of the family. The Peanut Brigade was a different organization all together. In 1976, Time magazine referred to her as a “tiny, stylishly dressed, white-haired dynamo” and the secret weapon of Carter’s campaign. Dolvin served as the coordinator of docents for the Carter Center from its inception in 1986.
Dolvin’s personal home, the W.J. Dolvin house, is located next to Bulloch Hall in the Historic District of Roswell and is often referred to as “President Jimmy Carter’s Roswell White House”. It is one of the few examples of late-Victorian architecture in that city.
An elder of the Roswell Presbyterian Church, Dolvin also founded the Refuge Resettlement Ministry. In 1992, Dolvin married a second time to Hubert “Hu” B. Visscher and became Mrs. Emily Dolvin-Visscher. She was named to the list of Roswell’s 15 Most Remarkable Citizens in 2004. Dolvin died at age 94 of congestive heart failure at her Roswell home on December 2, 2006.