According to the Encyclopedia of Sports Medicine and Science, men reach their physical prime around age 20 and leave their physical prime around age 35.
Writing for the Encyclopedia of Sports Medicine and Science, Roy J. Shepherd reports that maximal oxygen consumption begins to decline after age 25. Oxygen consumption is important in overall fitness and muscle development. Muscle strength tends to peak at age 25 and then plateau until age 35. Between 35 and 45 years old, men are still considered to be young middle-aged. After 45 years old, a man may be considered past his prime.
If "prime" is defined by performance in athletics, the top gymnasts are adolescents. Performance in aerobic events usually peaks in the mid-20's. Performance in athletic events, where skill and experience are important, tends to peak in the mid-30's.