arises from the tendency for researchers
and editors to handle experimental results that are positive (they found something) differently from results that are negative (found that something did not
happen) or inconclusive.
- "Publication bias occurs when the publication of research results depends on their nature and direction.
Positive results bias, a type of publication bias, occurs when authors are more likely to submit, or editors accept, positive than null (negative or inconclusive) results. A related term, "the file drawer problem
", refers to the tendency for those negative or inconclusive results to remain hidden and unpublished.
Even a small number of studies lost in the file drawer can
result in a significant bias..
Outcome reporting bias occurs when several outcomes within a trial are measured but these are reported selectively depending on the strength and direction of those results.
A related term that has been coined is HARKing (Hypothesizing After the Results are Known).
For example, skeptics often argue that there is (or at least was) a strong publication bias in the field of parapsychology, leading to a File drawer problem.
Suppose that several studies about the influence of power lines on cancer are performed. They are admitted for publication only if they show a correlation with a 95% confidence level. If only the positive results make it to publication, because negative results are just shelved, we do not know how many studies were performed, so it is possible that all the published results are type I errors -- studies which mistakenly showed a correlation when in truth there is none.
Effect on meta-analysis
The effect of this is that published studies may not be truly representative of all valid studies undertaken, and this bias
may distort meta-analyses
and systematic reviews
of large numbers of studies - on which evidence-based medicine
, for example, increasingly relies. The problem may be particularly significant when the research is sponsored by entities that may have a financial interest in achieving favourable results.
Those undertaking meta-analyses and systematic reviews need to take account of publication bias in the methods they use for identifying the studies to include in the review. Among other techniques to minimise the effects of publication bias, they may need to perform a thorough search for unpublished studies, and to use such analytical tools as a Begg's funnel plot or Egger's plot to quantify the potential presence of publication bias. Tests for publications bias rely on the underlying theory that small studies with small sample size (and large variance) would be more prone to publication bias, while large-scale studies would be less likely to escape public knowledge and more likely to be published regardless of significance of findings. Thus, when overall estimates are plotted against the variance (sample size), a symmetrical funnel is usually formed in the absence of publication bias, while a skewed asymmetrical funnel is observed in presence of potential publication bias.
Extending the funnel plot, the "Trim and Fill" method has also been suggested as a method to infer the existence of unpublished hidden studies, as determined from a funnel plot, and subsequently correct the meta-analysis by imputing the presence of missing studies to yield an unbiased pooled estimate.
Examples of Publication Bias
One study compared Chinese and non-Chinese studies of gene-disease associations and found that "Chinese studies in general reported a stronger gene-disease association and more frequently a statistically significant result.
One possible interpretation of this result is selective publication (publication bias).
Risks and Remedies
According to researcher John Ionnidis, negative papers are most likely to be suppressed:
- when the studies conducted in a field are smaller
- when effect sizes are smaller
- when there is a greater number and lesser preselection of tested relationships
- where there is greater flexibility in designs, definitions, outcomes, and analytical modes
- when there is greater financial and other interest and prejudice
- when more teams are involved in a scientific field in chase of statistical significance.
Ionnidis further asserts that "Claimed Research Findings May Often Be Simply Accurate Measures of the Prevailing Bias"
Ionnidis' remedies include:
- Better powered studies
Enhanced research standards including
- Low-bias Meta-Analysis
- Large studies where they can be expected to very definitive results or test major, general concepts
Considering, before running an experiment, what they believe the chances are that they are testing a true or non-true relationship.
- Pre-registration of protocols (as for randomized trials)
- Registration or networking of data collections within fields (as in fields where researchers are expected to generate hypotheses after collecting data)
- Adopting from randomized controlled trials the principles of developing and adhering to a protocol.
- Properly assessing the false positive report probability based on the statistical power of the test
- Reconfirming (whenever ethically acceptable) established findings of "classic" studies, using large studies designed with minimal bias
In September 2004
, editors of several prominent medical journals (including the New England Journal of Medicine
, The Lancet
, Annals of Internal Medicine
, and JAMA
) announced that they would no longer publish results of drug research sponsored by pharmaceutical companies unless that research was registered in a public database from the start.
In this way, negative results should no longer be able to disappear.