He had a high profile career in the civil service from 1961 to 1998, serving as Private Secretary to five prime ministers. He was Secretary of the Cabinet and Head of the Home Civil Service from 1988 to 1998.
Butler was educated at Harrow School in London and University College, Oxford, where he took a double first in Mods and Greats and twice gained a Rugby Blue. He joined HM Treasury in 1961, becoming Private Secretary to the Financial Secretary to the Treasury 1964–66 and Secretary to the Budget Committee 1965–69.
Early in his career he was occasionally confused with his namesake Rab Butler. Memos for Rab Butler, some highly sensitive, ended up on his desk, and some of his ended up on Rab's. It was agreed that all memos ambiguously addressed to "R Butler" should go to Rab's office first, and then Rab's office would send on any intended for the other R Butler. It is said that one day the young Butler, who was still playing first class rugby, received a letter that read: "You have been selected for the Harlequins 1st XV on Saturday. Please be at Twickenham by 2 p.m.". Underneath, in Rab's distinctive handwriting, was the message: "Dear Robin, I am not free on Saturday. Please could you deputise for me? Rab."
In 1969, he was seconded to the Bank of England and several City institutions. Later at HM Treasury as Assistant Secretary, General Expenditure Intelligence Division, he led the team which installed the UK Government's computerised financial information system 1975–77. He had been a founder member of the Central Policy Review Staff under Lord Rothschild 1971–2. After several senior appointments at the Treasury, he became second Permanent Secretary, Public Expenditure, 1985–7.
He was Private Secretary to Prime Ministers Edward Heath (1972–74) and Harold Wilson (1974–75), and Principal Private Secretary to Margaret Thatcher (1982–85). He was also Cabinet Secretary during the premierships of Margaret Thatcher, John Major and Tony Blair.
Butler married Gillian Lois Galley in 1962, known as Gill, and they remain happily married to this day. They have a son and two daughters.
He became the non-executive Director of HSBC Group from 1998 to 2008. He is also Chairman of the Corporate Sustainability Committee and the HSBC Global Education Trust.
In February 2004 it was announced that Lord Butler of Brockwell would chair an inquiry into the use of intelligence in the lead up to the 2003 Iraq War. His report (widely known as the "Butler Report") concluded that some of the intelligence about Iraq's possession of Weapons of Mass Destruction was seriously flawed.