statistical mirages · metaphor 13 of 100
The treatment helps men. It helps women. It hurts people. All three sentences describe the same data. Before you argue policy from a trend, ask who chose the grouping — because the grouping chose the trend.
Two newspapers cover the same hospital study. One runs the tables by patient group and reports, accurately, that the new treatment outperformed the old in every group. The other runs the totals and reports, just as accurately, that patients who got the new treatment did worse. Neither reporter lied. Neither slipped a decimal. They divided honestly and printed what the division said — and the two headlines contradict each other flatly.
The instinct is to assume someone must be wrong, because arithmetic is supposed to be neutral. But aggregation is not neutral. Pooling groups of different sizes and different baselines can reverse every within-group truth at once. Berkeley's 1973 admissions numbers, wage-gap statistics, school comparisons, and real treatment decisions all live inside this trap.
Default counts are the classic kidney-stone study (Charig et al., BMJ 1986). Every rate on this panel is success ÷ patients, computed live from the counts you see — nothing is faked. The slider moves patients between the two treatments while holding each cell's success rate fixed (to the nearest whole patient), so only the mix changes.
52 fixed points, drawn once from a seeded generator so the page is reproducible; dragging moves a whole cluster rigidly. All three lines are ordinary least-squares fits recomputed live from the visible points.
The grouping did it
Inside each group, the comparison is fair: like patients against like patients. The pooled number is a different thing wearing the same units — a weighted average, where each treatment's overall rate is dragged toward the groups it happened to treat most. In the default data, treatment A took on mostly severe cases, whose baseline success is low for everyone; treatment B coasted on mostly mild ones. Pool them, and A's genuine edge is buried under its harder caseload. The totals aren't comparing treatments anymore. They're comparing caseloads.
So the reversal needs exactly two ingredients: groups with different baselines, and treatments with different mixes of those groups. The variable doing both jobs — steering who gets which treatment and setting how hard success is — is the confounder. Kill either ingredient and the mirage evaporates, which is precisely what the case-mix slider lets you do: it changes nothing about how well either treatment works, only who was sent where, and that alone flips the headline.
What to try
Which answer is true
No amount of extra data settles it. Both computations are correct; each is the true answer to a different question. The tiebreaker lives outside the table entirely — it is the causal question: what does the treatment do to a person? If severity influenced who got which treatment — doctors reaching for the stronger option when the case looked bad — then severity is a confounder, the within-group comparison is the one that answers the doing-question, and the pooled number is the mirage.
The working rule: group by the things that chose the treatment, never by things the treatment produced. Severity existed before any prescription was written, so splitting by it is legitimate. But a grouping variable that sits downstream of the choice — a side effect, a symptom the treatment itself altered — poisons the comparison instead of purifying it. The table can't tell you which situation you're in. Only a story about the world can, which is why Simpson's own 1951 paper already conceded that the "sensible interpretation" depends on facts the numbers don't contain.
The rhetoric of aggregation
In 1973, Berkeley's pooled graduate admissions looked plainly biased against women. Department by department, the tilt mostly ran the other way: women had applied, in larger numbers, to the departments that rejected nearly everyone. Both facts sat in one dataset, waiting for someone to choose a level to quote — and every wage gap, school ranking, and district comparison offers its partisans the same menu. The move is rarely fraud. It's selection among true sentences.
Which suggests the honest posture, for writers and readers alike: show both levels, then argue openly about the confounder — about what steered people into their groups — instead of quietly quoting the level that flatters. And when a trend is wielded at you, two questions before any rebuttal: who chose this grouping? and what decided who landed in which group? The first question has an author. The second has an answer.
The mapping
| Mathematics | Life |
|---|---|
| the groups | The hidden strata running through any population — severity, department, era, neighborhood. |
| within-group trend | What is true of people who are genuinely like each other. |
| pooled trend | What is true of the crowd as a piece of accounting — real, but answering a different question. |
| the confounder | Whatever decides who lands in which group before anything is measured. |
| the reversal | How two honest reporters read one dataset into opposite headlines. |
| choosing the partition | Choosing the conclusion — the quiet editorial act inside every "the data show." |
Where the metaphor tears
This page kept whispering "trust the groups" — but if the grouping variable sits downstream of the treatment (a side effect, a blood measurement the drug itself moved), then splitting by it is the error, and the pooled comparison is the causal one. The paradox cuts both ways, and nothing in the arithmetic announces which way. Only causal knowledge — which arrows point where — can say. Disaggregating is not a virtue; it's a bet about the world.
The instrument gives you one labeled confounder and one dial. Real populations are crossed by many overlapping partitions at once — severity and age and site and era — and adjusting for all of them shreds the data into cells too small to say anything. Worse, no variable arrives wearing a badge that says "confounder." The lab makes visible a structure that, in the wild, must be argued for.
Played carelessly, this metaphor breeds nihilism — every inconvenient number dismissed with "well, Simpson's paradox." But nothing here showed statistics lying; it showed two true answers to two different questions. The actual lesson is narrower and more demanding: state your grouping, defend it causally, and show the other level. Distrust of every number is just the mirror image of credulity toward one.