Castille played college football for the Alabama Crimson Tide and was on the last team coached by the legendary Bear Bryant. Castille was a pallbearer at Bryant's funeral on January 28, 1983. Castille played for Alabama from 1979-1982, recording a school record 16 interceptions and making the College Football All-America Team in 1982. In the 1982 Liberty Bowl at the end of the season, Castille intercepted three passes in a 21-15 win and was selected as the game's MVP.
While playing for the Broncos during the 1987 AFC Championship Game versus the Cleveland Browns, Castille will forever be remembered for stripping the ball from Browns' running back Earnest Byner in a play called "The Fumble." He also recorded an interception in Super Bowl XXII, although this is not as widely remembered because his team ended up losing 42-10.