Generally, the most challenging math course offered in a compulsory secondary education setting is Advanced Placement (AP) calculus. The course offerings between public and private schools may differ substantially, and many college preparatory schools may advance students further in specialized topics.
Beyond secondary education, there are a wide variety of studies of mathematics. University math students, whether undergraduate or post-baccalaureate, tend to specialize their topics.
The advantage of taking AP math courses in secondary education is that a sufficiently high score on the AP exam concluding the course can merit college course credits to be awarded. This allows advanced students to jump-start their post-secondary studies.