Trap/Statistical Bias/No. 0895
Selection Bias
You read glowing reviews of a difficult course, then notice that every reviewer made it to the final lesson.
- Evidence
- Well established
- Read
- 6 min
- Links
- 19 connections
01You've seen this when…
- in life
You compare two fitness programs using their before-and-after photos. Neither website shows the people who quit after three weeks.
- at work
An employee survey reports high morale. Your manager celebrates, but the most frustrated colleagues have already left and never received it.
- out in the world
A city reports strong results from a job-training program. Its follow-up calls reach people with steady phone numbers, while many struggling participants remain uncounted.
02The idea
The people in your data got there somehow. Someone invited them, they agreed to participate, they survived long enough to be counted, or they passed a filter. That route into the data can change the answer you get.
Selection bias occurs when the process deciding which cases are observed or analyzed distorts the comparison you want to make. The distortion may affect an average, a percentage, or a relationship between two things.
The key is the gap between the group you observed and the group your conclusion is about. Reviews from course graduates may describe graduates perfectly. Their usefulness for predicting how a new student will fare depends on how well graduates represent new students.
This isn’t just another name for a small sample. A million selected cases can give a very precise answer to the wrong question. With confounding, another factor influences the things you’re comparing. Selection bias comes from who enters or remains in the analysis, though both problems can occur together.
03Why it happens
- People choose whether to appear. Those who answer a survey, post a review, or volunteer for research may differ from those who don’t. If that difference matters to the question, participation changes the result. This is often nonresponse bias or self-selection.
- Cases disappear along the way. Participants drop out, including customers who cancel and patients who stop attending. An analysis of only those who remain can make results look better—or worse—than they are. Attrition bias concerns losses during follow-up; survivorship bias focuses on the survivors left visible.
- A gatekeeper selects cases. Hiring rules, hospital admissions, publication decisions, and platform moderation all determine what becomes observable. You may be studying the gatekeeper’s choices as much as the underlying population.
- A filter can manufacture a relationship. Suppose both technical skill and interview polish help someone get hired. Among hires, one strength may appear to compensate for the other, even if the two are unrelated among applicants. Selecting on their shared result can create collider bias.
04A worked example
Before the 1936 U.S. presidential election, The Literary Digest mailed roughly ten million ballots and received about 2.4 million replies. Its poll predicted a victory for Republican Alf Landon. Franklin Roosevelt instead won about 61% of the popular vote.
What it looks like An unusually convincing forecast. Millions of responses seem far more trustworthy than a modest survey. The sheer volume makes ordinary sampling error look like a minor concern.
What’s actually going on The mailing lists relied heavily on sources such as telephone directories and automobile-registration records. During the Depression, those sources did not cover voters evenly. Then another filter operated: recipients chose whether to reply. The returned ballots were not a random slice of the electorate.
The familiar explanation—that the poll simply reached too many affluent car and telephone owners—is incomplete. Peverill Squire’s later analysis found that nonresponse also mattered. The failure involved both who could receive a ballot and who sent one back. More replies from the same process could have made the wrong forecast look even more certain.
What would have helped Defining the electorate as the target, building a sampling plan that covered it, and checking how respondents differed from nonrespondents. The crucial investment was a better route from voters to data. Another million ballots from the same process would have preserved the selection problem.
05How to spot it
06What to do instead
- Name the target before reading the result. Write down who the conclusion is supposed to describe. Then compare that group with the cases actually analyzed.
- Trace the route into the dataset. Follow eligibility, invitation, participation, retention, and exclusions. A directed acyclic graph can help reveal whether a filter depends on both sides of a comparison.
- Account for every stage. Ask how many cases were eligible, contacted, enrolled, followed, and analyzed. Record why cases disappeared, not just how many.
- Look for missing groups directly. Seek outcomes for dropouts and nonrespondents, or compare participants with a broader registry. Don’t assume silence means success, failure, or indifference.
- Use adjustments only with defensible assumptions. Weighting can help when measured characteristics explain selection and relevant groups have a chance to appear. It cannot reliably reconstruct groups you never observe or fix selection driven by information you don’t have.
- Test how much the missing cases could change the answer. Use sensitivity analysis to examine plausible alternatives. If reasonable assumptions reverse the conclusion, report that uncertainty rather than hiding it.
07When it isn’t selection bias
A selected group can be appropriate for the question you’re asking. If you want to describe this year’s marathon finishers, measuring finishers is appropriate. The problem starts when you generalize their experience to everyone who entered—or everyone considering entering.
Selection can also be intentional and manageable. A survey may oversample a small demographic group so researchers can estimate its results more precisely, then use known sampling probabilities to calculate population estimates.
Terminology varies across fields. Sampling bias usually emphasizes recruitment or coverage; selection bias can also arise from later exclusions and follow-up losses. The boundary depends on the field. Understanding the mechanism matters more than winning an argument over the label.
Even a well-randomized trial can be distorted if outcomes are missing differently across treatment groups. A sound result for volunteers may have limited external validity for other people.
08Roots
In 1946, physician and statistician Joseph Berkson examined a deceptively convenient source of evidence: hospital records. The charts contained diagnoses, but the people behind them had first passed through an admission process. He showed that hospital selection could create associations between diseases that did not hold in the wider population. The hospital served as a collection site and a filter that helped shape the evidence.
Survey researchers had confronted the same basic difficulty from another direction. A mailing list covered a selected part of the public, and returned questionnaires came from a self-selected part of that list. The spectacular failure of The Literary Digest made the difference memorable, but concern about representative samples preceded that election. There was no single inventor of the broader idea.
Economists later brought selection into questions about work and wages. Pay could be observed for people who worked, while the decision to work was itself related to characteristics economists wanted to study. James Heckman’s influential 1979 paper formalized this problem and a correction under specific assumptions. Across surveys, medicine, and economics, the lesson converged: explaining the numbers requires explaining how those numbers became observable.
09How solid is this?
Selection bias is a mathematically understood mechanism with extensive evidence across surveys, epidemiology, and economics. A particular selection process need not cause substantial bias; its direction and size depend on how selection relates to the question. Corrections require assumptions that the observed data alone may not verify.
10Connections
- Often confused with Adverse Selection, Confounding, Sampling Bias
- Countered by Outside View vs. Inside View, Directed Acyclic Graph, Sensitivity Analysis
- Part of Correlation-Causation Fallacy
- Includes Cherry Picking, Nonresponse Bias, Attrition Bias, Survivorship Bias, Collider Bias
- See also Hot Hand Fallacy, Mean vs. Median, Regression to the Mean, Risk Compensation, External Validity, Missing Not at Random, Simpson’s Paradox
+ 9 more in the list
11Origin and sources
Developed across survey sampling, epidemiology, and econometrics, with no single inventor. Joseph Berkson described hospital-selection effects in 1946; James Heckman formalized sample-selection problems in economics in 1979.
- [1]Squire, P. (1988). Why the 1936 Literary Digest Poll Failed. Public Opinion Quarterly, 52(1), 125–133.
- [2]Berkson, J. (1946). Limitations of the Application of Fourfold Table Analysis to Hospital Data. Biometrics Bulletin, 2(3), 47–53.
- [3]Heckman, J. J. (1979). Sample Selection Bias as a Specification Error. Econometrica, 47(1), 153–161.
- [4]Hernán, M. A., Hernández-Díaz, S., & Robins, J. M. (2004). A Structural Approach to Selection Bias. Epidemiology, 15(5), 615–625.
Suggest an edit· Updated 2026-10-02