statistical mirages · metaphor 77 of 100
The dashboard says the two teams are identical: same average, same spread, same trend. One is a healthy group having an ordinary year; the other is ten quiet people and one crisis. Summaries are alibis — the same numbers can testify for wildly different realities.
In 1973 the statistician Francis Anscombe built four small datasets as a provocation. Same mean of x. Same mean of y. Same variance, same correlation, same fitted line, to two decimal places. But one is an honest linear cloud; one is a clean curve pretending to be a line; one is ten points in perfect order plus a single wrecking outlier; one is a vertical stack with a lone influential point inventing a trend single-handed. Every summary statistic swears they are the same. The eye needs half a second to know they are not.
He built them because his colleagues trusted tables and distrusted pictures. Fifty years later we run organizations, relationships, and countries off dashboards that would score all four of these identical — and would score your year, and a very different year, identical too. Below: the four testimonies, then a forger's bench where you can discover by hand exactly how little a dashboard pins down.
The quartet · four witnesses under oath
real data — Anscombe (1973), hard-coded verbatim; every statistic beneath each panel is computed live from its 11 points, not asserted. hover or focus a panel to enlarge; click one to load it onto the bench below.
The forger's bench · hide the truth from the dashboard
drag any point — all four statistics recompute live on every move. keyboard: focus the plot, arrows nudge the ringed point (shift = coarse), N selects the next point.
Four testimonies, one alibi
Set I is the world the statistics honestly describe: a linear tendency plus noise, the one case where the summary and the scene agree. Set II is a clean curve — a relationship so lawful it has no scatter at all — which the straight line reports as a middling trend with residuals. The correlation r can only testify about straightness; it files curvature under "unexplained." Set III is the cruelest: ten points lie on a perfect line, and one outlier drags the fit away from all of them. The line obeys the outlier and betrays the ten — least squares is a court where the loudest witness wins. Set IV has no relationship at all: ten identical x-values and one influential point at the far edge, single-handedly inventing a trend of 0.5. Remove it and there is nothing to summarize.
Four different mechanisms, four different stories, four different right responses — and one alibi: x̄ = 9.0, ȳ = 7.5, r = 0.816, ŷ = 3.0 + 0.5x. Anyone reading only the numbers would treat all four the same way. That is the operating condition of everyone who governs what they cannot see.
What to try
The dashboard organization
Every metric regime silently assumes its numbers arise the honest way — a real tendency plus ordinary noise. But suppose your direct report is having set III's year: ten fine weeks and one catastrophe. The average reads exactly like steady mediocrity, so the catastrophe is averaged into invisibility, its weight redistributed across weeks where nothing was wrong. The quarterly review then addresses a person who does not exist: the slightly-below-par plodder the summary invented. Meanwhile a set II career — someone lawfully accelerating along a curve — reads as the same tepid trend as everyone else.
And the forger's bench cuts the other way too: anyone managed by four numbers can learn to sculpt the eighteen dimensions the numbers don't watch. Goodhart's law is just this bench, played from the inside, on a salary.
The discipline of looking
Anscombe's own moral was procedural: make graphs before you trust numbers. The life version is the same discipline. Ask of every metric you rely on: which eighteen dimensions can this not see? Then go look at a few of them directly. The scatterplot has human equivalents — the site visit, the skip-level conversation, reading ten actual support tickets instead of the satisfaction score, sitting through one whole class instead of the test average. Looking is expensive, unscalable, and unrigorous-feeling, which is why it gets delegated to the summary. But the summary was never evidence about shape. It was four equations, politely silent about the other eighteen.
The test of a good dashboard is not whether it is accurate — all four panels above are accurately summarized. It is whether anyone still goes to the scene.
The mapping
| Mathematics | Life |
|---|---|
| the summary statistics | The dashboard, the GPA, the quarterly review — the few numbers allowed to speak for the whole. |
| the four scatters | The different lives, teams, and years that score identically on those numbers. |
| set III's outlier | The one crisis averaged into invisibility — absorbed by ten weeks where nothing was wrong. |
| degrees of freedom | How much reality a summary leaves unspecified: 22 dimensions of fact, 4 of them watched. |
| the forger's bench | How easily appearances are maintained at green — by accident, or by anyone gaming the metric. |
| plotting the data | Actually looking: at the case, the person, the week — the visit to the scene the numbers summarize. |
Where the metaphor tears
Summaries exist because nobody can eyeball ten thousand cases; abolishing the dashboard just replaces statistics with anecdote, which is a worse summary with better lighting. The honest fix is richer compression — quantiles, distributions, small multiples — plus sampled looking: a few scenes visited at random, so the summary knows it can be audited.
This page flatters vision, but the whole cluster it lives in testifies against the eye too: we see streaks in noise, trends in random walks, patterns in scatter that a permutation test would dissolve. "Just look" is not an epistemology. The plot corrects the summary; the statistics correct the plot; neither is allowed to testify alone.
A scatterplot of y against x already embeds a hypothesis about what matters; the crisis may live in a variable nobody thought to chart. Looking is not neutral access to reality — it is just far less compressed than four numbers, and honest about being a choice. The discipline is not "trust your eyes" but "keep more dimensions alive than your conclusions need."