Mental Models a field guide

Trap/Statistical Bias/No. 0797

Publication Bias

Publication bias occurs when a study’s result affects its chance of being published, making published research an uneven sample of all studies done. In statistics, Theodore Sterling and Robert Rosenthal described how this can hide null results and overstate the evidence for an effect.

a trap: easy to walk into
Evidence
Well established
Read
6 min
Links
13 connections
Useful when
Deciding under uncertainty · Evaluating a claim · Reading data and statistics

01You've seen this when…

  1. in life

    A meditation app links to published studies showing benefits. You read them before subscribing. Trials that found little improvement never appear on your screen.

  2. at work

    Your team chooses a sales-training method after finding four encouraging papers. Researchers who tried the same method without success have left their reports unfinished.

  3. out in the world

    A school district adopts a reading program backed by journal articles. When an evaluator checks a trial registry, several completed studies have no published results.

02The idea

A literature search begins with the studies that became visible. Other studies may have been completed, analyzed and quietly set aside. If the result helps determine which group a study enters, the visible evidence becomes a selected sample.

That is publication bias, a form of selection bias. Favorable, statistically significant or surprising results often have a better chance of reaching publication. Null results—findings that provide no clear evidence of the tested effect—can remain in researchers’ file drawers.

The consequence reaches beyond missing papers. Suppose twenty teams test a treatment with little benefit. Chance alone can give a few teams encouraging results. If those studies become the available literature, a reader sees a pattern that the full set of studies would weaken.

Two neighboring problems deserve separate names. Selective outcome reporting chooses which findings within a study to reveal. P-hacking uses flexible analysis choices to obtain a favorable statistical result. Publication bias concerns whether the study itself becomes available. These problems can reinforce one another, and each requires a different check.

03Why it happens

  • Encouraging findings seem easier to publish. Researchers may expect journals to reject an inconclusive result and never submit the manuscript. Editors and reviewers can also favor findings that seem novel or decisive. Selection can happen at several points between a completed analysis and a published paper.
  • Writing competes with the next project. Turning a study into a paper takes time. Researchers under pressure to build a publication record may prioritize promising findings and postpone disappointing ones indefinitely.
  • Some results threaten an interested party. Sponsors, institutions or researchers may have financial or reputational reasons to keep an unfavorable study out of view. The missing result can disappear through an explicit decision or through delays and neglected follow-up.
  • Statistical thresholds make chance findings attractive. Small studies produce imprecise estimates. Among many such studies, a few will show large effects by chance and cross a significance threshold. Preferentially publishing those results makes the available estimates look stronger.

A meta-analysis can combine published findings with great mathematical care and still inherit this selection. More papers improve precision around whatever evidence entered the analysis; they cannot supply the results that remain missing.

04A worked example

In 2008, Erick Turner and colleagues compared journal publications with US Food and Drug Administration records for 74 trials of 12 antidepressants. The regulatory records gave them a view of trials that a journal search alone would miss.

What it looks like An overwhelmingly favorable research literature. Among the trials represented in journal publications, 94% appeared positive.

What’s actually going on The FDA judged 51% of all 74 trials positive. Thirty-one percent of the trials were unpublished. Some trials judged negative or questionable by the FDA were published in ways that conveyed a positive outcome. The review therefore exposed both whole-study nonpublication and selective presentation of findings. Across the drugs, the pooled effect size calculated from journal reports was 32% larger than the effect size calculated from FDA records.

What would have helped Assembling the evidence from regulatory records as well as journals, identifying every trial before examining its result, and making completed results accessible regardless of outcome. These steps would have given readers a fuller estimate of benefit. The review showed inflated apparent efficacy; it did not establish that the drugs had no benefit.

05How to spot it

These are warning signs. Diagnosing publication bias requires evidence about how studies became available.

06What to do instead

  • Start with an inventory of studies. When the decision matters, look for a systematic review that searches beyond published papers. Compare registered studies with completed reports and record unexplained gaps.
  • Check the search and disclosure methods. A review should describe where it looked, how it handled unavailable results, and whether authors or sponsors supplied additional data. A claim that publication bias was tested needs enough detail to assess the test.
  • Use bias diagnostics cautiously. A funnel plot compares effect estimates with their precision. Asymmetry can suggest missing studies, but it has several possible causes. With fewer than ten studies, standard asymmetry tests generally have too little power to be dependable.
  • Build result-independent reporting into new research. Preregistration creates a record of the planned study. Registered reports allow journals to commit to publication after reviewing the question and methods, before the results are known, subject to their acceptance conditions. Publishing completed results in a registry also makes them accessible.

Preregistration helps expose missing studies, but registration alone cannot make their results appear. For an immediate decision, ask how the conclusion changes under plausible amounts of missing evidence. Large, precise independent replications can also reduce reliance on an encouraging early literature.

07Where it doesn’t settle the verdict

Publication bias can favor striking results in either direction. A dramatic finding of harm may attract attention too. The direction of distortion depends on which outcomes are rewarded or suppressed.

A null result can arise from a small effect, weak measurement or insufficient statistical power. Its interpretation depends on the study’s design and precision. Missing null findings still matter because they help reveal how uncertain the evidence is.

Bias-adjustment methods depend on assumptions about the missing studies. Their estimates are useful sensitivity checks; the unpublished findings remain unknown until someone recovers them. Conclusions that survive several plausible assumptions deserve more confidence.

08Roots

In 1959, statistician Theodore Sterling examined papers in psychology journals and noticed a remarkable pattern. Among the papers in his sample that used significance tests, 97.3% rejected the null hypothesis. He asked whether publication decisions could help explain that concentration of successful tests.

The question changed how a reader could interpret a shelf of journals. Each published paper might be competently conducted, while the collection gave a distorted picture because unsuccessful tests were less likely to reach print. Sterling described how that filtering could undermine inferences drawn from the published record.

Two decades later, psychologist Robert Rosenthal gave the missing research a memorable location: the file drawer. His 1979 paper explored how unpublished null results could change conclusions and how many such studies might be needed to weaken an apparently significant finding.

The problem later became central to evidence synthesis. Trial registries and regulatory records offered something Sterling’s journal survey lacked: a way to identify studies independently of whether their results appeared in print. Comparisons such as Turner’s antidepressant review made the missing evidence visible.

09How solid is this?

ContestedMixedUsefulEstablished

Comparisons between registered or regulatory study records and journal publications establish that results can affect publication. The size and direction of the distortion vary across fields; funnel-plot asymmetry and bias-adjustment methods cannot determine the missing evidence with certainty.

10Connections

confused withconfused withcountered bycountered bycountered bycountered bycountered bypart ofpart ofincludesPublicationBiasNot written yetSelectiveOutcome ReportingNot written yetP-HackingNot written yetMeta-AnalysisNot written yetSystematicReviewNot written yetPreregistrationNot written yetRegisteredReportNot written yetReplicationSelection BiasSurvivorshipBiasNot written yetFile-DrawerProblem

+ 3 more in the list

11Origin and sources

Theodore Sterling described result-dependent publication in 1959. Robert Rosenthal developed the closely related file-drawer problem in 1979.

  1. [1]Sterling, T. D. (1959). Publication Decisions and Their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa. Journal of the American Statistical Association, 54(285), 30–34.
  2. [2]Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641.
  3. [3]Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008). Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy. New England Journal of Medicine, 358(3), 252–260.
  4. [4]Sterne, J. A. C., et al. (2011). Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ, 343, d4002.

Suggest an edit· Updated 2026-10-02