Elmslie was born in Lethbridge, near Geelong, and although he had a secondary education, he followed his father's trade as a stonemason. He was employed on the first Wilson Hall at Melbourne University and on St Patrick's Cathedral. From 1888 he was an official of the Operative Stonemason's Society, and a delegate to the Melbourne Trades Hall. He was also President of the South Melbourne Football Club, ancestor of the Sydney Swans.
In 1898, Elmslie was one of the founders of the Victorian Labour Federation, which had as its object "the unification of the workers in one all-comprehensive and extensive union." In 1902 he was elected to the Victorian Legislative Assembly as Labor member for Albert Park. Labor in Victoria in the early federal period was much weaker than in the other states, partly because of the continuing attraction of Deakinite liberalism for many voters, partly because Victoria did not have the huge pastoral and mining areas that the other mainland states had. The Parliamentary Labor Party remained small and contained limited talent. Elmslie became Deputy Leader in 1912 and Leader in 1913.
At the 1911 election Labor won only 20 seats to the various factions of the Liberal Party's 43. But in December 1913 the Liberal Premier, William Watt resigned after a dispute with the rural faction of his own party. The acting Governor, Sir John Madden, surprised the Liberals by sending for Elmslie, who on 9 December formed Victoria's first Labor government.
Elmslie had no chance of a long tenure, or even of meeting the House as Premier, since under the law of the time ministers had to resign their seats and contest by-elections before they could take their seats. The Liberal factions re-united, and Watt moved a no-confidence motion in Elmslie, which Elmslie had to watch from the gallery since he was technically not a member. Elmslie was duly voted out and Watt resumed office on 22 December.
Elmslie remained as Labor leader until shortly before his death in 1918, although his health had broken down in 1916, requiring a long break. During World War I Elmslie supported the Allied cause but opposed conscription for overseas service. He died at his home in South Melbourne and was given a state funeral.
Elmslie was largely forgotten until members of the Labor Historical Graves Committee discovered his neglected grave in the Melbourne General Cemetery in the 1990s. A new memorial headstone over his grave was unveiled by Steve Bracks, Labor Premier of Victoria, on 9 March 2001.