Despite his later political longevity, Murray's early political career was marked by inability to get elected. He lost five consecutive elections at the federal and provincial level before finally winning a seat. Despite his electoral failures he was highly regarded within the Liberal Party and was nominated by Premier William Stevens Fielding to succeed him when Fielding left provincial politics in 1896 to join the federal cabinet of Sir Wilfrid Laurier. Murray was sworn in as premier and took a seat in the legislature when he was acclaimed as a candidate in Victoria County.
As premier, Murray was a practitioner of brokerage politics. His government continued the public works projects of his predecessor in the area of rail (doubling the tracks in the province within a decade), road and bridge construction. It also improved post-secondary education, particularly in the area of agriculture and vocational education in founding an agricultural college at Truro as well as the Nova Scotia Technical College.
The Murray government also introduced progressive labour legislation such as the Factories Act in 1908 and workman's compensation for injuries on the job in 1915. In the area of public health the Murray government appointed public health officers, establishing county health clinics and founded a research hospital for tuberculosis patients.
After almost three decades in power Murray retired from politics in January 1923. He twice declined the offer of knighthood and twice refused earlier offers to join the federal Cabinet of Sir Wilfrid Laurier. He died in Montreal.