Baker, born in Troy, New York, was the son of George Ellis Baker and Eveline Stevens. He married Florence Tucker Baker (daughter of Benjamin Franklin Baker and Sophronia J. Whitney) in 1869, and was the father of:
His son George Fisher Baker, Jr.'s daughter Edith married John M. Schiff of Kuhn, Loeb & Co.
In 1929 George Baker commissioned the construction of a one-of-a-kind Pierce-Arrow town car, built by LeBaron, for the wedding of his daughter. The cars uniquely high roof line (5" taller than standard models) was specifically requested by Mr. Baker who wished to wear his top hat while riding. At the time this car was finished, Mr. Baker was one of the 10 richest men in the United States and easily one of the wealthiest individuals in history up to that time.
The vehicle, having been owned by such a prominent figure in American history, has significant historical value. The car was rediscovered in 1978 in a barn in Ohio alongside a Rolls Royce Boat Tail Speedster formerly owned by Fred Astaire.
The unique features of the car reflect the affluence of its former owner. Trim lining the rear compartment of the car is made of 24 carat gold, as are the perfume dispensers and intercom.
The car is currently part of the White Glove Collection, located in La Crosse, Wisconsin.
The April 14, 1924, Edition of Time Magazine had this to say about Baker:
The March 26, 1934, Time magazine magazine article called Baker
A 1934 article in Newsweek describes him as one of the most imposing figures in banking history. In the November, 1994 issue of Worth magazine, in an interview with James Grant, editor of a financial newsletter, Baker is described as the hidebound turn-of-the-century banker who always got his loans repaid and was one of Grant's heroes.