"A population is said to be in an evolutionarily stable state
if its genetic composition is restored by selection after a disturbance, provided the disturbance is not too large. Such a population can be genetically monomorphic or polymorphic." --Maynard Smith (1982).
Contrast this with the definition of an evolutionarily stable strategy, which is a game theory concept.
"An ESS or evolutionarily stable strategy is a strategy such that, if all the members of a population adopt it, no mutant strategy can invade." --Maynard Smith (1982).
Evolutionarily stable states are frequently used to identify solutions to the replicator equation, given in its linear payoff form by
is said to be evolutionarily stable
if for all
in some neighborhood of
This is similar but distinct from the concept of an evolutionarily stable strategy.
It can be shown that for the replicator equation if a state is evolutionarily stable then it is an asymptotically stable rest point, so the evolutionarily stable states are often taken as solutions of the replicator equation. The converse is true if the game matrix is symmetric.
The concept of evolutionary stability is equivalent to the concept of strong stability for normal form games, due to Cressman.