Trap/Logical Fallacy/No. 1027
Texas Sharpshooter Fallacy
The Texas sharpshooter fallacy is selecting a pattern after seeing data and treating it as if it confirmed a prior prediction. In statistics, this ignores the other patterns that could have been chosen, inflating false positives. Unlike cherry picking, it tailors the claim to the evidence.
- Evidence
- Well established
- Read
- 6 min
- Links
- 9 connections
- Useful when
- Designing products · Evaluating a claim · Forecasting · Reading data and statistics
01You've seen this when…
- in life
You inspect a month of sleep records, comparing dinners, workouts, bedtimes, and supplements. Herbal tea lines up with your best nights, so you decide it’s the secret.
- at work
A campaign shows no overall improvement. After splitting results by city, device, age, and purchase history, the team finds one promising group and puts it at the center of the presentation.
- out in the world
Someone draws a boundary around several cancer cases on a map. The boundary skips nearby streets with no cases, and the resulting cluster is presented as evidence against a local factory.
02The idea
A pattern means something different when you choose it before looking than when you choose it because it looks impressive.
Suppose you predict that a particular customer group will respond to a campaign, then test that prediction. A strong result supports a specific claim. Now suppose you inspect dozens of groups and choose whichever responds best. That group’s result has been selected partly for being unusually good. Treating it as the outcome of a single planned test leaves out the search that produced it.
The name comes from a story about a gunman who fires at a barn, then paints a target around the tightest cluster of bullet holes. His accuracy is excellent only because he gets to choose the target afterward.
Finding the cluster isn’t the fallacy. Presenting it as independent confirmation is. Exploration can generate valuable ideas. But the data that suggested an idea usually cannot also provide an ordinary, unadjusted test of it.
Cherry picking selects evidence favorable to a claim while ignoring contrary evidence. In the Texas sharpshooter fallacy, the claim itself is tailored to the evidence. The clustering illusion describes how random clusters can tempt us to see meaning. The Texas sharpshooter fallacy concerns how we select and justify a pattern, regardless of whether it ultimately proves real.
03Why it happens
- Every additional search gives chance another opening. Try enough customer segments, time windows, map boundaries, or outcomes, and something will look unusual. Under idealized conditions, 20 independent tests with no real effects and a 5% false-alarm rate each have about a 64% chance of producing at least one false alarm. Real searches often overlap, so that calculation isn’t universal. The principle is the multiple comparisons problem.
- The successful search hides the unsuccessful ones. A presentation shows the striking chart, not the seventeen charts that looked ordinary. Readers judge the result as if it were the only question asked.
- A good explanation makes hindsight feel like foresight. Once returning customers on mobile look responsive, it’s easy to explain why they should love the campaign. The explanation may be plausible without having predicted anything.
- The search need not be deliberate. You can make reasonable choices after seeing the data: exclude an unusual month, change an age boundary, or examine another outcome. Together, those choices create a garden of forking paths. No conscious attempt to manufacture a result is required.
Confirmation bias can steer the search toward a favored explanation. But even someone with no preferred answer can mistake the most interesting result for strong evidence.
04A worked example
Peter Austin and colleagues set out to demonstrate the hazards of testing many hypotheses in a 2006 study using Ontario health records. They searched for associations between astrological signs and many hospital diagnoses in one portion of the data, then checked findings in a separate validation sample.
What it looks like Some birth signs appear to carry elevated risks for particular medical conditions. Presented alone, those findings could look like evidence that astrology predicts health.
What’s actually going on The researchers have given chance many opportunities to produce an association. The particular sign-and-diagnosis combinations become interesting because they stand out in the first dataset. The apparently meaningful findings did not survive the study’s validation and multiple-testing checks. Reporting only the initial associations would paint the target around the hits.
What would have helped The researchers used safeguards that a misleading account would omit: checking separate data and accounting for the many comparisons. Their study demonstrates the correction by showing that the apparent support for astrology fails those checks. The lesson is that even a large dataset can supply convincing-looking coincidences when the search is broad enough.
05How to spot it
06What to do instead
- Separate discovery from testing. Label a newly noticed pattern as a lead. Specify what it predicts before collecting or opening the next dataset.
- Write down the main test in advance. Record the outcome, comparison, time window, and important exclusions. Formal preregistration makes this distinction visible to other people, but a dated plan also helps your own thinking.
- Count the search behind the result. Ask how many outcomes, subgroups, boundaries, and analyses could have produced an impressive finding. For formal statistical claims, use methods appropriate to the full search that produced each result.
- Keep some evidence untouched. Use one portion to explore and another to test the selected claim. Cross-validation can help with predictive models, provided you don’t repeatedly tune decisions against the supposedly untouched results.
- Look for a new check. A later period, another location, or an independent replication can test whether the pattern travels. Different conditions may change a real effect, so interpret failures with those conditions in mind.
- Report what changed during analysis. Readers should be able to distinguish planned questions from discoveries. An exploratory result retains its value when labeled honestly.
07When it isn’t a fallacy
Looking for patterns is a legitimate part of learning. An engineer may notice that failures concentrate after a software update. A doctor may notice an unusual combination of symptoms. Neither needs to pretend the observation was predicted.
The distinction is between a reason to investigate and a completed demonstration. A discovery can become convincing through fresh evidence, a credible mechanism, or statistical methods that account for selection. A preregistered experiment is one of several ways to support a useful conclusion.
A pattern can still be true when someone notices it afterward. Criticism of the reasoning and disproof of the conclusion are different claims. The right response is to lower the confidence warranted by the original analysis and ask for a better test.
08Roots
In 1977, statistician John Tukey published Exploratory Data Analysis, a book built around looking closely at data with simple plots and summaries. He wanted analysts to find features that a rigid, preselected analysis might miss. The challenge was to make room for discovery without confusing it with confirmation.
The barn-wall story gives that distinction a memorable picture: bullet holes first, painted bull’s-eye second. It circulates as a statistical joke and later as the name of a logical fallacy. Its exact naming origin is uncertain; there is no securely established inventor or first use to credit here. The underlying statistical concern is older than Tukey’s book.
Modern research gives the gunman many more targets. Software makes it easy to test hundreds of outcomes and slices of data. Work on undisclosed analytical flexibility, including Joseph Simmons, Leif Nelson, and Uri Simonsohn’s 2011 paper, shows how ordinary choices can generate misleading statistical significance. The metaphor remains useful because it makes the missing question visible: what other targets could have been painted?
09How solid is this?
Selecting findings after inspecting data can inflate false-positive rates and exaggerate apparent effects unless the analysis accounts for selection. This is mathematically established and demonstrated empirically; an individual discovered pattern may still be real.
10Connections
- Often confused with Cherry Picking, Clustering Illusion
- Countered byPreregistration, Cross-Validation, Replication
- Can follow from Confirmation Bias
- Part ofMultiple Comparisons Problem
- See alsoGarden of Forking Paths, HARKing
11Origin and sources
A popular statistical metaphor whose exact naming origin is uncertain. John Tukey’s Exploratory Data Analysis (1977) helped clarify the distinction between discovering patterns and confirming claims, but he is not credited here with naming the fallacy.
- [1]Austin, P. C., Mamdani, M. M., Juurlink, D. N., & Hux, J. E. (2006). Testing multiple statistical hypotheses resulted in spurious associations: a study of astrological signs and health. Journal of Clinical Epidemiology, 59(9), 964–969.
- [2]Tukey, J. W. (1977). Exploratory Data Analysis. Addison-Wesley.
- [3]Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science, 22(11), 1359–1366.
Suggest an edit· Updated 2026-10-02