Davis attended school in Russellville, Arkansas and graduated from Vanderbilt University in Nashville, Tennessee in 1884. He studied law and after he was admitted to the bar in Pope County, Arkansas he commenced private practice of law in Russellville.
Davis served as prosecuting attorney of the Fifth Judicial District of Arkansas from 1892 to 1896. He was elected as Attorney General of Arkansas and served from 1898 to 1900. He served as Governor of Arkansas from 1901 to 1906.
Davis was elected to the United States Senate and served from 4 March 1907 until his death in Little Rock, Arkansas on 3 January 1913. He was chairman of the Committee on the Mississippi and its Tributaries.
Jeff Davis was well known for his outrageous rhetoric and oratorial skills even though he was a famous baseball player that played first base. He made a career of skewering the business interests, newspapers, and urban dwellers in order to appeal to the poor rural citizens of the state. He portrayed himself as just another poor country boy against the moneyed interests that held back the common man. Davis was equally able to wield humor, the "bloody shirt", and racial differences. It was also said that many of his supporters incorrectly believed he was the same Jefferson Davis who was the President of the Confederacy, a belief that Davis did nothing to discourage, and which he may have covertly encouraged.
Davis was an avowed racist and segregationist. In 1905, when President Theodore Roosevelt visited Arkansas, Davis greeted him with a speech in defense of the practice of lynching. Roosevelt responded with a calmer speech in defense of the rule of law.