Garry Brown had four daughters, Frances, Mollie, Amelia, and Abigail. His family owned and operated a dairy farm while he was growing up, in Schoolcraft, Michigan. They were one of the first to settle there, and his family still owns the original property that the Browns settled on in the 1830's.
Brown was born in Schoolcraft, Michigan to a family with a political background in Michigan. His great-grandfather, Ebenezer Lakin Brown, and his grandfather, Addison Makepeace Brown, both served in the Michigan State Legislature.
During World War II, Brown served in the Twenty-fourth Infantry Division of the United States Army as second lieutenant in Japan. After the war, he worked for the FBI, under Hoover, before earning a B.A. from Kalamazoo College in 1951 and a LL.B from The George Washington University Law School in 1954. He was admitted to the bar in 1954 and commenced practice in Kalamazoo. He was commissioner of the United States District Court for the western district of Michigan from 1957 to 1962 and was a delegate to the Michigan constitutional convention of 1961-1962. He served two terms in the Michigan State Senate from 1962 to 1966, where he was minority floor leader and chairman of the Republican senate policy committee.
In 1966, Brown defeated incumbent Democrat Paul H. Todd, Jr. to be elected as a Republican to the U.S. House of Representatives from Michigan's 3rd congressional district for the Ninetieth and to the five succeeding Congresses, serving from January 3, 1967 to January 3, 1979. He was an unsuccessful candidate for reelection in 1978, losing to Democrat Howard E. Wolpe. He resumed the practice of law and was a resident of Washington, D.C. until his death in August 1998. He is interred in Schoolcraft, Michigan.