Scott played for Huntingdon Boys and St Neots Town before joining the groundstaff at West Ham United in 1957. After three years, and having made 12 appearances for England Youth, Scott made his senior debut on 6 February 1960 against Chelsea.
Partnering Phil Woosnam on the right wing, Scott played 97 games for West Ham, scoring 19 goals, before moving to Aston Villa. He played his last game for West Ham on 11 September 1965 against Leicester City.
Scott joined Villa for £25,000 and rekindled his partnership with Woosnam. He went on to make 57 appearances and score five goals for the Midlanders until his departure in September 1967. He then joined Torquay United, where he played under former Hammer Frank O'Farrell and linked up with his old team-mate John Bond.
Scott followed Bond, who went on to become manager of Bournemouth, in July 1970. Bournemouth, who also had ex-Hammer Ken Brown as coach, finished runners-up in Scott's first season. He made a total of 61 League appearances, scoring six goals. He moved to Exeter City in June 1972 and managed two goals in 51 League appearances before being forced to retire though injury in May 1974.