Fisher was elected to the House of Commons for Fulham in 1885, a seat he held until 1906, and again from 1910 to 1918. In 1896 he was appointed a Junior Lord of the Treasury in the Conservative administration of Lord Salisbury, a post he held until 1902, and then served under Arthur Balfour as Financial Secretary to the Treasury from 1902 to 1903. He was to remain out of office for the next twelve years but was admitted to the Privy Council in 1911. He returned to the government as Parliamentary Secretary to the Local Government Board in May 1915 in the newly formed coalition government led by H. H. Asquith. He retained this post also when David Lloyd George became Prime Minister in December 1916, but in June 1917 he was promoted to President of the Local Government Board with a seat in the cabinet. In November 1918 he was made Chancellor of the Duchy of Lancaster and Minister of Information, and a few days later he was raised to the peerage as Baron Downham, of Fulham in the County of London. He stepped down from the government already in January 1919.
Lord Downham died in July 1920. His title died with him.