Smit went to Pretoria Boys High School where he was head prefect in 1996 and played in the first rugby team from 1994 to 1996.
Smit played his first Springbok game in 2000 at the age of 22, when South Africa beat Canada 51-18 at Basil Kenyon Stadium in East London. He was on and off of the reserve bench until 2004 when he was made captain of the squad by then new coach Jake White.
As of 2005 Smit is one of South Africa's most successful captains ever, having lead the Springboks to victory in 16 of the 22 games that he captained, a win percentage of 72%. This makes him just as successful as other South African captains such as Gary Teichmann and Francois Pienaar.
Smit's leadership qualities came to the fore in the 2007 Rugby World Cup, his calm under pressure and inspirational captaincy seeing South Africa win the William Web Ellis Trophy for the second time. Smit will also lead South Africa in the 2008 Tri Nations against Australia and New Zealand. In the first game against New Zealand in Wellington, he was spear tackled by New Zealand lock Brad Thorn who was suspended for one game, He now will miss the rest of the 2008 Tri-Nations because of a grion injury and veteran Springboks lock, Victor Matfield will lead the side.
During the run-up to the 2007 Rugby World Cup, he was linked with the French club Clermont. He eventually signed a two-year deal with Clermont, effective after the World Cup. Despite his move to France, the South African Rugby Union announced on 20 February 2008 that he would retain his Boks captaincy. . After just one year with Clermont John Smit resigned with The Natal Sharks and the Super 14 side Sharks