While Brosnan was a brilliant railroader and innovator, his ham-fisted tactics of management, specifically during the takeover of the Central of Georgia Railroad in 1963, forever marked him as a ruthless tyrant to many. For many years after Brosnan was gone one could not find any former Central of Georgia (and very few Southern) agreement employees who would speak of him in printable language. Nearly 1500 employees of the Central were terminated without cause on Brosnan's orders after the merger. Four years later, a Federal Court ruled that the Southern had to pay retroactive pay to these unfairly terminated workers.
In 1967, Brosnan was informed by the Southern Board of Directors that he would have to step down as Chairman and CEO of the Southern Railway.
He remained on the Board of directors of the Southern until 1983. Brosnan died in 1985.