According to the University of North Carolina School of Medicine, most males stop growing by the time they reach 20. The same is true of females.
While growing stops, changes in the skeletal system do not. According to the UNC School of Medicine, the pelvis continues widening between the ages of 20 and 79, and may contribute to weight gain in later life. A Duke University webpage states that the human skull continues growing after puberty as well. The changes to the skull involve forward movement of the forehead and backward movement of the cheekbones. As a result, a man's face subtly changes as he grows older.