Group selection was used as a popular explanation for adaptations, especially by V. C. Wynne-Edwards. For several decades, however, critiques, particularly by George C. Williams, John Maynard Smith and C.M. Perrins (1964), cast serious doubt on group selection as a major mechanism of evolution, and though some scientists have pursued the idea over the last few decades, only recently have group selection models seen a minor resurgence (albeit not as a fundamental mechanism but as a phenomenon emergent from standard selection ).
However, theoretical models of the 1960s seemed to imply that the effect of group selection was negligible. Alleles are likely to be held on a population-wide level, leaving nothing for group selection to select for. Additionally, generation time is much longer for groups than it is for individuals. Assuming conflicting selection pressures, individual selection will occur much faster, swamping any changes potentially favored by group selection. The Price equation can partition variance caused by natural selection at the individual level and the group level, and individual level selection generally causes greater effects.
Experimental results starting in the late 1970s demonstrated that group selection was far more effective than the then-current theoretical models had predicted. A review of this experimental work has shown that the early group selection models were flawed because they assumed that genes acted independently, whereas in the experimental work it was apparent that gene interaction, and more importantly, genetically based interactions among individuals, were an important source of the response to group selection (e.g. ). As a result many are beginning to recognize that group selection, or more appropriately multilevel selection, is potentially an important force in evolution.
More recently, Yaneer Bar-Yam has claimed that the gene-centered view (and thus Ronald Fisher's treatment of evolution) relies upon a mathematical approximation that is not generally valid. Bar-Yam argues that the approximation is a dynamic form of the Mean Field approximation frequently used in physics and whose limitations are recognized there. In biology, the approximation breaks down when there are spatial populations resulting in inhomogeneous genetic types (called symmetry breaking in physics). Such symmetry breaking may also correspond to speciation.
Spatial populations of predators and prey have also been shown to show restraint of reproduction at equilibrium, both individually and through social communication, as originally proposed by Wynne-Edwards. While these spatial populations do not have well-defined groups for group selection, the local spatial interactions of organisms in transient groups are sufficient to lead to a kind of multi-level selection. There is however as yet no evidence that these processes operate in the situations where Wynne-Edwards posited them; Rauch et al's analysis, for example, is of a host-parasite situation, which was recognised as one where group selection was possible even by E. O. Wilson (1975), in a treatise broadly hostile to the whole idea of group selection.
But though Smith gave a mathematical model by which group selection might work, he was skeptical that it would happen in nature often enough to be worth considering. His reasoning was that the specific conditions for group selection to take hold, namely the repeated isolation, mixture, and reisolation of organisms would be so rare and unlikely to occur in nature that it was almost certainly not a significant evolutionary force. In their 1999 books Unto Others, and in various articles before this, Sober and Wilson challenge this view. While one of their challenges takes the form of naming organisms, such as the so called “brain worm” (Dicrocoelium dendriticum) which has a life cycle very much like that of the haystack organisms mentioned above, they present a more significant argument based on the notion of trait groups. Trait groups can occur within larger groups through the interaction of particular genetic traits, and need not interact for a generation to promote survival value. Sober and Wilson see Kin selection, which is often considered an alternative to group selection, as a special case of a trait groups. To see how a trait group could be beneficial, lets imagine an altruist trait, such as cooperation with another organism even in such cases were it only benefits 40% as much as the organism it helps, and a selfish trait such as cooperating with another organism only when it will benefit more than the organism it helps. The first trait is considered altruistic in Sober and Wilson’s sense because the within-group fitness of the altruistic organism drops every time it cooperates compared with the other member of the group. Now imagine five organisms, one of which is altruistic in regards to this trait, and the rest of which are selfish. Assume that each case of cooperation increases the chance of survival and reproduction by 10 units which is divided among the interacting pair (group of two). Now assume that member of the population groups/interacts with each other member of the population one time. After all the interactions have taken place, the selfish organisms have each acquired 6 units. This is because they all refuse to cooperation with other selfish members (since it is impossible for both members to benefit more than the other), but each takes advantage of the altruist benefits over that individual in a ratio of 60% to 40%. The altruist on the other hand has interacted with 4 selfish organisms and thus has earned 16 units (four for each encounter) and thus has a greater survival advantage than the selfish members of the population. The altruist ends up winning the survival “war” even though it came out behind in every survival “battle”.
Because individuals can form hundreds or even thousands of trait groups within its life span, the trait group selection model does not have to rely on the unlikely situation of an entire population isolating into groups, merging, and then isolating into groups again. Likewise the rate at which trait groups can form and dissolve can be many times faster than the rate at which individuals reproduce, providing cumulative as opposed to all-or-nothing benefits. It is important to note that this argument has not settled the issue of group selection however. There is still heavy debate as to whether or not such formations count as “real” groups in the traditional biological sense of groups affected by group selection.
In recent years, the limitations of earlier models have been addressed, and newer models suggest that selection may sometimes act above the gene level. Recently David Sloan Wilson and Elliot Sober have argued that the case against group selection has been overstated. They focus their argument on whether groups can have functional organization in the same way individuals do and, consequently, if groups can also be "vehicles" for selection. For example, groups that cooperate better may have out-reproduced those which did not. Resurrected in this way, Wilson & Sober's new group selection is usually called multilevel selection theory.
Although Richard Dawkins and fellow advocates of the gene-centered view of evolution remain unconvinced (see, for example, ), Wilson & Sober's work has been part of a broad revival of interest in multilevel selection as an explanation for evolutionary phenomena. Indeed, in a 2005 article, E. O. Wilson (often regarded as the father of sociobiology) argued that kin selection could no longer be thought of as underlying the evolution of extreme sociality, for two reasons. First, some authors have shown that the argument that haplodiploid inheritance, characteristic of the Hymenoptera, creates a strong selection pressure towards nonreproductive castes is mathematically flawed (e.g. ). Secondly, eusociality no longer seems to be confined to the hymenopterans; increasing numbers of highly social taxa have been found in the years since Wilson's foundational text on sociobiology was published in 1975, including a variety of insect species, as well as a rodent species (the naked mole rat). Wilson suggests the equation for Hamilton's rule:
(where b represents the benefit to the recipient of altruism, c the cost to the altruist, and r their degree of relatedness) should be replaced by the more general equation
in which bk is the benefit to kin (b in the original equation) and be is the benefit accruing to the group as a whole. He then argues that, in the present state of the evidence in relation to social insects, it appears that be>>rbk, so that altruism needs to be explained in terms of selection at the colony level rather than at the kin level.