After a spell in the North American Soccer League with Washington Diplomats in 1978, Graydon began pursuing coaching ambitions.
Replacing colourful Dane, Jan Sorensen, Graydon's first task was to keep Walsall in the then Second Division, and to re-introduce an element of professionalism to the club. However, success went much further than this - with promotion being secured alongside then-England manager Kevin Keegan's big-spending Fulham side. What made the feat even more achievable was not only that Walsall were favourites for relegation in the summer of 1998, but that behind them lay the likes of Gillingham, Preston North End, Wigan Athletic and Manchester City - all having spent millions of pounds that season. Walsall's team cost approximately £80,000 with perhaps the smallest operating budget of any team in the league. Within the League Managers Association, Graydon polled only behind Sir Alex Ferguson, in the poll for Manager of the Season.
Relegation from the First Division followed the season after, although, despite early struggle, Graydon took his team to the brink of survival. This was no mean feat, and the odyssey to the First Division had allowed Graydon to re-assess what needed to be done to ensure that next time Walsall gained promotion, they would stay in the higher division.
Indeed, Graydon again worked his midas touch the following season, taking Walsall to the Division Two play-off final - defeating Reading 3-2 after extra time to, yet again, earn promotion. In a true sign of the gentleman Graydon had become known as, instead of celebrating immediately, he embraced with Reading manager Alan Pardew, and shook the hand of each Reading player, including former Walsall captain, Adrian Viveash.
Graydon's second spell in the First Division was just as unsuccessful as the first. After a dismal performance in the January Black Country derby with West Bromwich Albion, Graydon was sacked. This decision was met with derision amongst the footballing press; however, the same press expected Graydon to walk into another job in football within days.
Graydon gladly accepted the chance to save his boy-hood favourites from relegation. However, the same strict methods that Graydon had employed to success at the Bescot Stadium were not so successful at Rovers, with relegation avoided yet again the following season. Graydon was sacked in January 2004, and is yet to return to management.
Since leaving Bristol Rovers, Graydon accompanied Howard Wilkinson to manage in China during 2004. This arrangement lasted a matter of months, however.
In February 2006, Graydon was approached by Leicester City manager Rob Kelly, and became the club's first team coach. After the end of the season, he left the club.