Brassey was the fourth son of the railway contractor Thomas Brassey and his wife Maria, daughter of Thomas Harrison. The Liberal MP Thomas Brassey, 1st Earl Brassey, and Henry Brassey were his elder brothers and Henry Brassey, 1st Baron Brassey of Apethorpe, his nephew. He was educated at Eton and University College, Oxford. He was a Lieutenant in the 14th Hussars and a Colonel in the Queen's Own Oxfordshire Hussars and served as High Sheriff of Oxfordshire in 1878. In 1895 he entered Parliament for Banbury, a seat he held until 1906. He lived at Charlton Park, Gloucestershire, Heythorn Hall, Chipping Norton, Oxfordshire, and at 29 Berkeley Square, London.
Brassey married the Hon. Maria Matilda Helena, daughter of John Charles Robert Bingham, 4th Baron Clanmorris, in 1871. They had three sons and five daughters. Their eldest son Robert also represented Banbury in the House of Commons very briefly in 1910. He died in January 1918, aged 73. His wife, who was appointed a OBE in 1919, died in July 1943.