In 1836 Rankine began to study a spectrum of scientific topics at the University of Edinburgh, including natural history under Robert Jameson and natural philosophy under James David Forbes. Under Forbes he was awarded prizes for essays on methods of physical inquiry and on the undulatory (or wave) theory of light. During vacations, he assisted his father who, from 1830, was manager and, later, effective treasurer and engineer of the Edinburgh and Dalkeith Railway which brought coal into the growing city. He left the University of Edinburgh in 1838 without a degree (which was not then unusual) and, perhaps because of straitened family finances, became an apprentice to Sir John Benjamin Macneill, who was at the time surveyor to the Irish Railway Commission. During his pupilage he developed a technique, later known as Rankine's method, for laying out railway curves, fully exploiting the theodolite and making a substantial improvement in accuracy and productivity over existing methods. In fact, the technique was simultaneously in use by other engineers - and in the 1860s there was a minor dispute about Rankine's priority.
The year 1842 also marked Rankine's first attempt to reduce the phenomena of heat to a mathematical form but he was frustrated by his lack of experimental data. At the time of Queen Victoria's visit to Scotland, he organised a large bonfire situated on Arthur's Seat, constructed with radiating air passages under the fuel. The bonfire served as a beacon to initiate a chain of other bonfires across Scotland.
Undaunted, he returned to his youthful fascination with the mechanics of the heat engine. Though his theory of circulating streams of elastic vortices whose volumes spontaneously adapted to their environment sounds fanciful to scientists formed on a modern account, by 1849, he had succeeded in finding the relationship between saturated vapour pressure and temperature. The following year, he used his theory to establish relationships between the temperature, pressure and density of gases, and expressions for the latent heat of evaporation of a liquid. He accurately predicted the surprising fact that the apparent specific heat of saturated steam would be negative.
Emboldened by his success, he set out to calculate the efficiency of heat engines and used his theory as a basis to deduce the principle, that the maximum efficiency of a heat engine is a function only of the two temperatures between which it operates. Though a similar result had already been derived by Rudolf Clausius and William Thomson, 1st Baron Kelvin, Rankine claimed that his result rested upon his hypothesis of molecular vortices alone, rather than upon Carnot's theory or some other additional assumption. The work marked the first step on Rankine's journey to develop a more complete theory of heat.
Rankine later recast the results of his molecular theories in terms of a macroscopic account of energy and its transformations. He defined and distinguished between actual energy which was lost in dynamic processes and potential energy by which it was replaced. He assumed the sum of the two energies to be constant, an idea already, although surely not for very long, familiar in the law of conservation of energy. From 1854, he made wide use of his thermodynamic function which he later realised was identical to the entropy of Clausius. By 1855, Rankine had formulated a science of energetics which gave an account of dynamics in terms of energy and its transformations rather than force and motion. The theory was very influential in the 1890s.
Energetics offered Rankine an alternative, and rather more mainstream, approach, to his science and, from the mid 1850s, he made rather less use of his molecular vortices. Yet he still claimed that Maxwell's work on electromagnetics was effectively an extension of his model. And, in 1864, he contended that the microscopic theories of heat proposed by Clausius and James Clerk Maxwell, based on linear atomic motion, were inadequate. It was only in 1869 that Rankine admitted the success of these rival theories. By that time, his own model of the atom had become almost identical with that of Thomson.
As was his constant aim, especially as a teacher of engineering, he used his own theories to develop a number of practical results and to elucidate their physical principles including:
The history of rotordynamics is replete with the interplay of theory and practice. Rankine first performed an analysis of a spinning shaft in 1869, but his model was not adequate and he predicted that supercritical speeds could not be attained.
Rankine was one of the first engineers to recognise that fatigue failures of railway axles was caused by the initiation and growth of brittle cracks. In the early 1840s he examined several broken axles, and showed that the axles had failed by progressive growth of a brittle crack from a shoulder or stress concentration on the shaft. It was likely that the front axle of one of the locos involved in the Versailles train crash failed in a similar way. Rankine presented his conclusions in a paper delivered to the Institution of Civil Engineers. His work was ignored however, by many engineers who persisted in believing that stress could cause "re-crystallisation" of the metal. The theory of recrystallisation was quite wrong, and inhibited worthwhile research until the work of William Fairbairn a few years later.
Rankine was instrumental in the formation of the 2nd Lanarkshire Rifle Volunteer Corps at Glasgow University in July 1859, becoming Major in 1860 after it was formed into the first company of the 2nd Battalion, 1st Lanarkshire Rifle Volunteer Corps; he served until 1864, when he resigned due to pressure of work - much of it associated with Naval Architecture.