Trap/Statistical Bias/No. 0877
Sampling Bias
Sampling bias occurs when a sample’s selection process systematically favors some parts of a population over others. In survey theory, advanced by Jerzy Neyman and others, it can skew estimates of the whole because the people or cases included differ from those left out.
Also called Sample Bias
- Evidence
- Well established
- Read
- 6 min
- Links
- 12 connections
01You've seen this when…
- in life
You ask your running club how often people exercise. Their answers make four workouts a week look normal for adults in your neighborhood.
- at work
A product team interviews ten customers who booked onboarding calls. Everyone praises the setup, and the team concludes new users find it easy.
- out in the world
A city collects comments on a bus redesign through its website. The report describes what residents want, though riders without easy internet access have little chance to contribute.
02The idea
A sample is a set of people or cases used to learn about a larger group. That larger group is the target population: everyone whose behavior, experience or views the conclusion is supposed to describe.
Sampling bias enters when the route into the sample systematically favors certain parts of that population. A customer survey sent to active accounts gives former customers no chance to appear. An interview conducted at a station at noon reaches a different mix of passengers from one conducted during the morning commute.
These differences can distort a result when they track the outcome being measured. Former customers may have more complaints. Midday passengers may have different travel needs. The analysis then carries the sample’s imbalance into its estimate of the population.
Sampling bias is a form of selection bias. Random sampling error is the ordinary fluctuation between samples drawn by chance. Increasing sample size can reduce that fluctuation while leaving a systematic selection problem intact.
03Why it happens
- The starting list has gaps. Researchers call the list or source from which cases can be selected the sampling frame. A list of registered users leaves out people who tried the service and never registered. A telephone directory leaves out people without listed numbers.
- Convenience controls recruitment. The easiest people to reach share a location, schedule or relationship with the researcher. Interviewing whoever walks past a booth turns foot traffic and time of day into selection rules.
- Participation attracts particular people. Strong opinions, spare time and interest in the topic can all affect who volunteers. Volunteer bias makes an open invitation especially risky for estimating how common a view is.
- Cases disappear along the way. Some invitees never reply, creating a risk of nonresponse bias. Failed businesses disappear from directories; dissatisfied users leave a platform. Studying only the remaining cases introduces survivorship bias.
Each step can change who appears in the final dataset. A sound initial draw still needs careful follow-through.
04A worked example
In 1936, The Literary Digest mailed roughly ten million ballots to predict the U.S. presidential election. Its recipient lists drew partly on telephone directories and automobile registration lists. About 2.4 million ballots came back. The forecast gave Republican Alf Landon about 57% of the vote. Franklin D. Roosevelt actually won about 61%.
What it looks like An unusually strong forecast. Millions of responses seem capable of overwhelming the uncertainty that troubles a smaller poll.
What’s actually going on The poll had two selection stages. First came the choice of recipients. Telephone and automobile lists gave people with those possessions easier access to the poll, during a period when household resources varied sharply. Then came the decision to return a ballot. Recipients with different voting preferences could respond at different rates.
Peverill Squire’s later analysis found that both selection and nonresponse contributed to the failure. The familiar explanation that the poll simply reached wealthy Republicans leaves out an important part of the evidence. The huge response count could offer little protection against either source of distortion.
What would have helped Drawing recipients through a probability-sampling design that covered the electorate more fully, following up with people who initially failed to respond, and checking the responding sample against reliable population information. Each measure would address a different point where the sample could become skewed.
05How to spot it
06What to do instead
- Define the population before collecting data. Specify the people, place and time period. A survey of current paying customers answers a different question from a survey of everyone who tried the product this year.
- Trace every step into the dataset. Record the source list, invitations, responses, dropouts and exclusions. At each step, identify who has a lower chance of appearing and whether that difference could affect the result.
- Use a probability sample when feasible. Give population members known, nonzero chances of selection. For a population with important subgroups, randomly select within those groups so a convenient channel cannot dominate recruitment.
- Make participation accessible. Use suitable languages, channels and times. Follow up with initial nonrespondents. These measures expand access; their success still needs checking against the population being studied.
- Use weights with explicit assumptions. Weighting can account for known selection probabilities or measured imbalances against reliable population benchmarks. Matching age or location does not establish that respondents and missing people share the same views within those groups.
When important groups remain absent, use sensitivity analysis: examine how the conclusion changes under plausible assumptions about their answers. If those assumptions determine the result, make that limitation prominent.
Keep the claim within the sample’s reach. Reporting what responding customers said gives readers a clear basis for judging its external validity, or how far the finding can travel.
07When it isn’t sampling bias
A probability sample can have an unusual mix purely by chance. That fluctuation belongs to random sampling error. The law of large numbers describes convergence under appropriate sampling conditions; a growing pile of conveniently selected observations does not automatically satisfy them.
Researchers also deliberately oversample small groups so they can study them precisely. Appropriate weighting can account for their unequal selection chances when estimating population totals or averages.
Representativeness depends on the question. A sample of marathon runners can support a study of marathon training. Extending its conclusions to all adults requires additional support. Even a visibly unbalanced sample may estimate a particular outcome accurately if the imbalance has little relationship to that outcome.
08Roots
In 1934, Polish statistician Jerzy Neyman brought a practical dispute into a paper for the Royal Statistical Society. Survey researchers wanted to learn about whole populations without counting everyone. One tempting approach was purposive selection: experts chose units that seemed representative. A few carefully chosen towns could resemble a miniature country on known measures.
The difficulty lay in everything those measures left unseen. Matching a national average on one characteristic gave no general assurance about another. Neyman’s paper compared purposive selection with stratified random sampling, where researchers divided a population into groups and selected randomly within them. This made selection explicit and supplied a mathematical basis for estimating sampling error.
The concern about unrepresentative samples was already older than the paper. Neyman’s contribution helped give survey practice a rigorous alternative to expert judgment alone. Two years later, the Literary Digest election failure made the stakes vivid to a much wider audience. The same question now follows online polls, customer interviews and administrative datasets: what process gave these cases their chance to appear?
09How solid is this?
Sampling theory formally establishes how selection processes can bias population estimates, and survey research documents these problems in practice. The size and direction depend on the outcome; a low response rate or demographic imbalance alone does not establish substantial bias.
10Connections
- Often confused with Selection Bias
- Countered by Sensitivity Analysis
- Can lead to Distribution Shift, External Validity
- Includes Survivorship Bias, Nonresponse Bias, Volunteer Bias
- See also Availability Heuristic, Conditional Probability, False Consensus Effect, Law of Large Numbers, Measurement Error
+ 2 more in the list
11Origin and sources
An older problem in survey practice. Jerzy Neyman’s 1934 work on stratified random sampling and purposive selection helped establish the theoretical basis for addressing it through probability sampling.
- [1]Neyman, J. (1934). On the Two Different Aspects of the Representative Method: The Method of Stratified Sampling and the Method of Purposive Selection. Journal of the Royal Statistical Society, 97(4), 558–625.
- [2]Squire, P. (1988). Why the 1936 Literary Digest Poll Failed. Public Opinion Quarterly, 52(1), 125–133.
- [3]Groves, R. M. (2006). Nonresponse Rates and Nonresponse Bias in Household Surveys. Public Opinion Quarterly, 70(5), 646–675.
- [4]Cochran, W. G. (1977). Sampling Techniques. 3rd ed. John Wiley & Sons.
Suggest an edit· Updated 2026-10-02