Baillie was a friend of Benjamin Disraeli, and in 1835 was actually called upon by Disraeli to serve as his second (after d'Orsay declined), when it appeared that Disraeli and Morgan O'Connell, the son of Daniel O'Connell, were going to fight a duel, which apparently did not actually occur.
In 1840 Baillie was elected as a Member of Parliament for Inverness-shire, and retained that seat until 1868. In the early 1840s he was associated with the notorious "Young England" movement, of which Disraeli was the head. Another member of that group, George Smythe, was Baillie's brother-in-law. He apparently broke with Sir Robert Peel over the Corn Laws and accepted minor office in Lord Derby's 1852 government as Joint Secretary to the Board of Control.
He died at the age of 82.