Kavanagh started 2006 playing in the Irish non-league for Wayside Celtic, playing in the team which was knocked out in the FAI Cup First Round by Intermediate side Killester United. He signed for UCD in July on amateur terms and scored five minutes into his debut as a substitute against Waterford United. His season was interrupted as he had already booked holidays before signing for UCD; however, he got a run in the side towards the end of the season and scored three goals in UCD's last three games to help the Students to sixth place in the league.
He went on trial to Birmingham City shortly afterwards, scoring in a reserve team game before making his move permanent in December.