Publication bias arises from the tendency for researchers and editors to handle experimental results that are positive (they found something) differently from results that are negative (found that something did not happen) or inconclusive.

Definition

"Publication bias occurs when the publication of research results depends on their nature and direction.

Positive results bias, a type of publication bias, occurs when authors are more likely to submit, or editors accept, positive than null (negative or inconclusive) results. A related term, "the file drawer problem", refers to the tendency for those negative or inconclusive results to remain hidden and unpublished. Even a small number of studies lost in the file drawer can result in a significant bias..

Outcome reporting bias occurs when several outcomes within a trial are measured but these are reported selectively depending on the strength and direction of those results. A related term that has been coined is HARKing (Hypothesizing After the Results are Known).

For example, skeptics often argue that there is (or at least was) a strong publication bias in the field of parapsychology, leading to a File drawer problem.

Illustration

Suppose that several studies about the influence of power lines on cancer are performed. They are admitted for publication only if they show a correlation with a 95% confidence level. If only the positive results make it to publication, because negative results are just shelved, we do not know how many studies were performed, so it is possible that all the published results are type I errors -- studies which mistakenly showed a correlation when in truth there is none.

Effect on meta-analysis

The effect of this is that published studies may not be truly representative of all valid studies undertaken, and this bias may distort meta-analyses and systematic reviews of large numbers of studies - on which evidence-based medicine, for example, increasingly relies. The problem may be particularly significant when the research is sponsored by entities that may have a financial interest in achieving favourable results.

Those undertaking meta-analyses and systematic reviews need to take account of publication bias in the methods they use for identifying the studies to include in the review. Among other techniques to minimise the effects of publication bias, they may need to perform a thorough search for unpublished studies, and to use such analytical tools as a Begg's funnel plot or Egger's plot to quantify the potential presence of publication bias. Tests for publications bias rely on the underlying theory that small studies with small sample size (and large variance) would be more prone to publication bias, while large-scale studies would be less likely to escape public knowledge and more likely to be published regardless of significance of findings. Thus, when overall estimates are plotted against the variance (sample size), a symmetrical funnel is usually formed in the absence of publication bias, while a skewed asymmetrical funnel is observed in presence of potential publication bias.

Extending the funnel plot, the "Trim and Fill" method has also been suggested as a method to infer the existence of unpublished hidden studies, as determined from a funnel plot, and subsequently correct the meta-analysis by imputing the presence of missing studies to yield an unbiased pooled estimate.

Examples of Publication Bias

One study compared Chinese and non-Chinese studies of gene-disease associations and found that "Chinese studies in general reported a stronger gene-disease association and more frequently a statistically significant result. One possible interpretation of this result is selective publication (publication bias).

Risks and Remedies

Risks

According to researcher John Ionnidis, negative papers are most likely to be suppressed:

1. when the studies conducted in a field are smaller
2. when effect sizes are smaller
3. when there is a greater number and lesser preselection of tested relationships
4. where there is greater flexibility in designs, definitions, outcomes, and analytical modes
5. when there is greater financial and other interest and prejudice
6. when more teams are involved in a scientific field in chase of statistical significance.

Ionnidis further asserts that "Claimed Research Findings May Often Be Simply Accurate Measures of the Prevailing Bias"

Remedies

Ionnidis' remedies include:

1. Better powered studies
• Low-bias Meta-Analysis
• Large studies where they can be expected to very definitive results or test major, general concepts
2. Enhanced research standards including
• Pre-registration of protocols (as for randomized trials)
• Registration or networking of data collections within fields (as in fields where researchers are expected to generate hypotheses after collecting data)
• Adopting from randomized controlled trials the principles of developing and adhering to a protocol.
3. Considering, before running an experiment, what they believe the chances are that they are testing a true or non-true relationship.
• Properly assessing the false positive report probability based on the statistical power of the test
• Reconfirming (whenever ethically acceptable) established findings of "classic" studies, using large studies designed with minimal bias

Study registration

In September 2004, editors of several prominent medical journals (including the New England Journal of Medicine, The Lancet, Annals of Internal Medicine, and JAMA) announced that they would no longer publish results of drug research sponsored by pharmaceutical companies unless that research was registered in a public database from the start. In this way, negative results should no longer be able to disappear.

See also

• Selection bias
• Confirmation bias
• File drawer problem
• List of cognitive biases
• Null hypothesis
• Parapsychology

External links

• Skeptic's Dictionary: positive outcome bias
• Skeptic's Dictionary: file-drawer effect
• Journal of Negative Results in Biomedicine

References