He was raised in Indian Hills, Kentucky, a suburb of Louisville, and has moved back to the area as an adult. His grandfather, C. Paul Downard, worked for a retail coal company, and served as president of the former Louisville Board of Aldermen. Downard's father, Norman, was president of a coal mining and mining equipment company, P&D Coal Mining Co. Inc.
Downard graduated from St. Xavier High School in 1964, also graduated from the University of Dayton School of Business Administration and earned a Master of Business Administration from Bellarmine University. A Roman Catholic, Downard and his wife Denise have two children.
Downard's 2006 Mayoral campaign was an underdog one, Abramson was a long-time mayor of the city and had wide name recognition, and most polls showed his approval rating at around 80%. Downard ran his campaign based on promises to improve the local economy and increase funding for public safety, saying that under Ambramson the city had slipped in both areas and was lagging behind nearby cities like Cincinnati and Indianapolis.
Downward was defeated in the election, receiving 76,423 votes (31%) to Abramson's 164,414 (67%). Although Downward was endorsed by police officers and firefighters, the Courier-Journal reported that his message that Abramson was soft on crime and economic issues failed to reach voters, saturated by other ads of, among other things, five congressional races in the Louisville television market. Nevertheless, Downward was arguably the most serious challenger Abramson had yet faced in the mayoral race.