living together · metaphor 41 of 100
The test scores rose and the children learned less. The KPIs were all green and the company was dying. When a measure becomes a target, it ceases to be a good measure — by a mechanism you can watch run.
The measure was never a scam. It was chosen because it correlated with the goal: test scores tracked learning, arrests tracked safety, steps tracked health. Somebody looked at the numbers, saw the relationship, and reasonably concluded that raising the number would raise the thing. The catch is quieter than fraud: the correlation lived in a world where nobody was optimizing the measure.
Aim at the number, and behavior pours into exactly the gap between proxy and goal — teaching to the test, arresting the arrestable, jiggling the wrist until the step counter believes you walked. The correlation was a photograph of an unpressured world; pressure is precisely what a target adds. Below is a small world where you supply the pressure yourself.
the photograph and the pressure
In the instrument, the true goal is a direction, g, and the metric is another direction, p, at an angle θ to it — the metric's honest imperfection. With pressure at zero, agents wander on their own mixed motives, which point roughly but not exactly at g. Read the correlation: it's high. Score on the proxy genuinely predicts contribution to the goal, because nothing is pushing on either. This is the world in which the metric was validated, and everything the manager believed about it was true — there.
Now raise λ. At mild pressure something seductive happens first: the proxy rises and the goal rises too, a little less — targeting genuinely focuses scattered effort, which is why targets get adopted. But keep pushing. Movement swings onto p itself; agents discover the slack in how the number is counted; the component of behavior orthogonal to g grows without bound. Proxy soars, goal stalls and sags, and the correlation — the entire justification for the metric — collapses toward zero and below. The metric didn't break. It was consumed. The act of using it destroyed the property that made it worth using.
what to try
the gap is where behavior goes
Why does behavior drift orthogonal to the goal rather than along it? Price per point. Real teaching raises test scores, but slowly, through the goal; test-prep tricks raise them directly, through the slack in how the test is scored. Once the score is what's paid for, the efficient move — for perfectly rational, not especially wicked agents — is to buy points wherever they're cheapest, and the gap between proxy and goal is exactly where points are cheapest. Teaching to the test is optimization, applied to the thing actually rewarded.
The same arithmetic runs far from school. Virtue signaling is targeting the legible part of virtue: the visible donation over the quiet one, the public apology over the private repair — effort sliding from goodness toward goodness-as-scored. Arrest quotas fill with the arrestable rather than the dangerous. Word counts fill with words. Watch the "gap, accumulated" counter in the instrument: that running total is real effort, honestly spent, that the goal never receives. Nobody stole it. The scoreboard simply pointed it elsewhere.
measurement without targeting
The law suggests its own countermeasures, and each one appears in the instrument as a way of keeping λ below the collapse. Audit random samples instead of scoring everyone: measurement survives because no individual can profit by gaming it. Keep some metrics unannounced or rotate criteria without warning: agents can't aim at an angle they can't see, so the honeymoon becomes permanent. Run low-stakes dashboards — numbers people look at but are not paid on — which is the slack dial turned high: you keep most of the information because you declined most of the pressure.
And then the honest limit: sometimes pressure is the point. A target exists to move behavior, and a metric nobody optimizes changes nothing. Every organization therefore holds a correlation budget — a stock of proxy-goal alignment that targeting spends down. The design question is where to spend: put full pressure on the few metrics with the smallest θ, keep the rest as unrewarded instruments, and treat any green dashboard that has carried heavy stakes for years with the suspicion it has earned.
the mapping
| Mathematics | Life |
|---|---|
| goal direction g | What you actually want — learning, safety, health — the thing that cannot be read off directly. |
| proxy direction p | What you can count: scores, arrests, steps, words. Countable because it is only part of the thing. |
| angle θ | The metric's honest imperfection — how far the countable sits from the wanted, even before anyone games it. |
| pressure λ | How hard careers, budgets, and bonuses hang on the number. |
| orthogonal drift | Gaming — real effort, honestly spent, that the goal never receives: test prep, quota arrests, wrist jiggles. |
| collapsing r | The law itself: the targeted measure stops measuring, precisely because it was targeted. |
where the metaphor tears
Manheim and Garrabrant distinguish regressional Goodhart (the proxy was always partly noise, so its extremes disappoint), extremal (the proxy–goal relationship changes shape far from the calibration range), causal (intervening on the proxy never moved the goal at all), and adversarial (agents deliberately exploit the gap). The instrument runs mainly the adversarial and extremal engines. A measure can fail you with no gaming anywhere — pure selection on a noisy proxy is enough — and that quieter failure needs different remedies than anti-gaming design.
The law is not nihilism about measurement. Set θ near zero in the instrument and pressure barely hurts: when the proxy nearly is the goal, there is almost no orthogonal space for behavior to leak into. "Lines of code" fails in weeks; "customer renewals" degrades far more slowly, because faking a renewal is nearly as hard as earning one. The craft of metric design — choosing proxies whose cheap points and true points coincide — is precisely the craft of shrinking θ, and it is worth real money.
Abolish the numbers and evaluation does not become holistic and wise; it becomes whoever the boss likes, whoever interviews well, whoever resembles the evaluator. Legible metrics were adopted partly as protection against exactly that, and for the least-connected people they still are. The lens prices metrics — it tells you what targeting costs and where the cost lands. It does not license throwing away the scoreboard and trusting the vibes of the powerful.