statistical mirages · metaphor 78 of 100
Ask one question of the noise and it stays silent. Ask twenty and it answers once — guaranteed, on schedule, with a straight face. False gods are manufactured by search volume, and the manufacture is invisible because the nineteen silences are never reported.
There is a famous cartoon about this — xkcd's jellybean study. Scientists test whether jellybeans cause acne. They don't. But a rumor insists it's only one color, so the scientists dutifully test purple, brown, pink, blue… twenty colors, twenty honest experiments. Green comes back significant, p < 0.05, and the front page duly reports: green jellybeans linked to acne — only 5% chance of coincidence! Nothing was faked. No one lied. Every test was run exactly by the book.
The book just says, quietly, that an honest test at the 5% level is fooled one time in twenty. Run twenty of them on pure nothing and you should expect one glowing false yes — and a 64% chance of at least one. The sin is not in any single test. The sin is in the asking, and in the forgetting of the asking. Below is a machine that manufactures false gods to order. Every p-value in it is real.
Instrument · the question machine
Every cell is one honest experiment run on pure noise — two groups of 40 that differ in nothing. Gold means p fell below the bar. α is fixed at 0.05.
honest computation: each cell is a real two-sample t-test (n = 40 per group, df = 78) on freshly drawn Gaussian noise; p-values via the exact t distribution. nothing is scripted to be significant.
The arithmetic of asking
One test at α = 0.05 is a fine instrument: when there is nothing to find, it stays silent 95% of the time. But silence doesn't compound the way you'd hope. Ask m independent questions of pure noise and the chance that all of them stay silent is 0.95ᵐ — which decays relentlessly. At five questions, a 23% chance of a false yes. At twenty, 64%. At a hundred, better than 99% — with about five false gods expected, each arriving with an authentic-looking p-value and a plausible story already forming around it.
Now add the second mechanism, the one that makes the first invisible: only the hits travel. The significant cell becomes a headline; the silent cells go into the drawer. The world never sees the denominator. A literature — or a memory — built this way isn't lying at any single point. It is a truthful record of a biased selection, which is a more durable kind of falsehood, because there's no error to catch. Toggle between the publish panel and the drawer above: same run, same data — one of them is a miracle, the other is arithmetic.
The audition
The machine works on people, too. Audition 100 coin-flippers, each calling 10 fair flips. Someone must score highest — and the best of a hundred guessers is reliably impressive, because you selected them for being impressive. The winner gets the spotlight and the breathless bio. The other ninety-nine were the drawer all along.
Instrument · the psychic audition
Press the button. Someone in the crowd is about to turn out to be remarkable.
honest computation: 1,000 real coin flips per audition; expected maximum computed exactly from the binomial(10, ½) distribution.
The bio always writes itself. Whoever wins, their childhood turns out to have been full of signs. Selection first, story second — and the story arrives so fast it feels like it was there all along. Rerun the audition: a different destiny, every time, on schedule.
What to try
The searchlight in your own life
The diet that finally "worked" — in the month you also started sleeping earlier, walking more, and drinking less, eight simultaneous experiments of which exactly one got the credit. The lucky shirt, distilled from a season of noticed-and-forgotten trials: the wins wearing it remembered, the losses wearing it quietly filed in the drawer. The relative who "predicted" the divorce — one confirmed warning drawn from her standing inventory of a hundred, the other ninety-nine expiring unrecorded. None of this requires self-deception in any single moment. It requires only what attention does naturally: ask constantly, and count nothing.
These are private multiple comparisons, run uncorrected for decades. The searchlight sweeps, something eventually glints, and the sweeping itself is instantly forgotten — so the glint looks like a discovery instead of a quota being filled.
The honest disciplines
Pre-registration: name your one question before the world can volunteer answers — the instrument's quietest toggle, and its most powerful. Correction: if you must ask twenty questions, raise the bar each must clear (Bonferroni's α/m is the bluntest version); you'll be fooled less and convinced less, and both are the point. Out-of-sample confirmation: whatever the searchlight found, make it perform again on fresh noise it hasn't met — green jellybeans, tested alone, next month. And the plainest discipline of all, the one that costs nothing and is almost never practiced: count your questions. Including the ones your attention asked without telling you. A finding means little until you know how large the family of attempts it was born into — the number the headline never carries, and the drawer always knows.
The mapping
| Mathematics | Life |
|---|---|
| a hypothesis test | Any question asked of noisy experience — does the shirt work, did the diet do it, was the warning prophetic. |
| α = 0.05 | How easily one question is fooled: even honest attention says a false yes one time in twenty. |
| m | How many questions you actually asked — including the ones your attention ran without telling you. |
| 1 − (1 − α)ᵐ | The family-wise near-certainty that something answers, once the asking is wide enough. |
| the file drawer | The unreported silences — the nineteen quiet tests whose absence is what makes the twentieth look like a miracle. |
| pre-registration | Naming your one question before the world can volunteer answers you didn't ask for. |
Where the metaphor tears
Genuine discoveries are also rare events, and a bar set at α/100 buries real effects along with false ones — the instrument shows you exactly this when the planted truth vanishes under Bonferroni at large m. Statisticians soften the blow (false-discovery-rate methods trade a little error for a lot of power), but no threshold arbitrates truth. Replication does. The correction is a tourniquet; the confirmation on fresh data is the cure.
Searching widely is how hypotheses are found — Fleming wasn't pre-registered when the mold cleared the plate. Sweeping the searchlight across a hundred cells is legitimate, even essential. The sin is a single, specific act of forgetting: treating what the search found as what a test confirmed. Explore promiscuously; just remember which of the two you were doing when the light glinted.
"Probably just multiple comparisons" can become a lazy universal solvent, dissolving findings that were in fact pre-registered, corrected, and replicated. The dismissal borrows the lens's authority without doing its arithmetic: to invoke it honestly you must argue that many questions were asked — count them, or gesture credibly at the drawer. A skeptic who won't is running his own uncorrected search: over all findings, for the ones he'd rather not believe.