Frazier was the son of a judge and a native of Bledsoe County, Tennessee. He was graduated from the University of Tennessee in 1878 and admitted to the bar in 1881, commencing practice in Chattanooga. A prosperous attorney, he was elected governor in 1902 as a Democrat. There were further reforms in education during his term as governor, and he was elected to a second two-year term in 1904. However, he served only two months of it. When U.S. Senator William B. Bate died in office, he arranged to have himself appointed to the vacant seat, resigning as governor on March 21, 1905. He was succeeded as governor by Speaker of the Tennessee Senate John I. Cox. He served as U.S. Senator until 1911. Unable to convince the state legislature to grant him another term (U.S. Senators were chosen by state legislatures until the ratification of the Seventeenth Amendment), he returned to his Chattanooga law practice, dying in that city in 1937 and being buried in its Forest Hill Cemetery.

His son, James B. Frazier, Jr. was a United States Representative from Tennessee.