belief & evidence · metaphor 45 of 100
Your theory of people was formed on the people it now explains so well. Of course it fits them — they are its training data. The only honest test of a belief is its performance on a life it was never fitted to.
Ask the office cynic how he knows everyone is selfish, and he will cite, with real feeling, the very colleagues who taught him the theory. He is not lying. But the evidence and the belief grew from the same soil, and citing one for the other is bookkeeping fraud, however sincere — the same experiences entered the ledger twice, once as reason, once as confirmation.
Statisticians met the same problem — every model flatters itself on the data that shaped it — and solved it with a discipline so simple it sounds like a trick: hide part of what you know from yourself, and grade your beliefs only on what they never saw. They call it cross-validation.
toy world: y = 0.7·sin(2.2x + 0.4) + 0.25x + noise(σ = 0.22) · N = 40 · k = 5 folds · every curve on this page is a real least-squares fit, recomputed from the points — nothing is hand-drawn.
| month | theory | registered miss | revisionist's retold miss |
|---|---|---|---|
| No entries yet. Your theory will forecast next month's eight experiences before seeing them — the miss goes on record. The revisionist waits, then fits a fresh story after the fact. | |||
Grading on hidden folds
A theory graded on its own training data holds a rigged exam: it was built by bending toward those points, accidents included, so every accident it memorized counts as insight. The fold machine breaks the rigging by rotation. Split the forty experiences into five folds; five times over, fit the theory on four and grade it only on the one it never touched. Each experience sits on the jury exactly once — and never judges the theory it helped build.
The in-sample miss answers how well do I explain what I've seen? — it can only fall as the theory grows more flexible. The cross-validated miss answers how well do I greet what I haven't? — and past a point it turns and rises, because extra flexibility is spent memorizing noise the hidden fold refuses to repeat. The distance between them is self-flattery with a number attached: exactly how much of your feeling of understanding comes from grading yourself on your own past.
What to try
Drag flexibility from 0 to 10: the in-sample miss falls forever while the cross-validated miss turns and climbs. At d=10 the flattery gap is widest — the better it explains, the worse it predicts.
Press run the five folds and follow one fold: its points hollow out, the curve refits without them, then they return to be missed or met. The whiskers are the honest residue — the part of the theory that was never about the world.
Pick a theory, then run a whole year. Your registered record should roughly match what the folds promised — that is the point: cross-validation is a rehearsal of the future. The revisionist's column, meanwhile, stays near 0.00 forever.
Press resample the world. The truth underneath is unchanged; only the accidents differ. The sweet spot barely moves; the d=10 theory's beautiful details become unrecognizable. What resamples away was never knowledge.
The protocol
Nothing in the fold machine requires mathematics — only the willingness to separate the evidence that builds a belief from the evidence that grades it. The human versions are all around, and all rare. Write down what you expect from the meeting before walking in. Register your opinion of the defendant before the verdict, of the hire before the first quarter. Judge your theory of relationships not on the friends who inspired it — they are training folds; the theory is partly their portrait — but on the next friend, the one it has never met.
The journal above is the whole protocol in one object: a forecast must be on record before the outcome, or it is not a forecast. What makes the discipline expensive is the exposure — a registered prediction can be seen to miss; an explanation offered afterward cannot. Cross-validation is the decision to prefer a grade you can trust to a grade you can enjoy.
The revisionist
The uncomfortable part: the revisionist is not stupid, and not exactly dishonest. Each retrofit genuinely improves the story of the past — the updated account really does fit the record better than the old one did. From inside, that is indistinguishable from learning. The feeling of insight is real; what's missing is that none of those ever-better stories was ever at risk. The journal is the only witness that the improvements never cashed forward — that the retold misses near zero were purchased after the outcomes were in hand.
Institutions run the same two columns. The post-hoc strategy deck explains last year with perfect fluency — every downturn its epicycle, every win its foresight — while the pre-registered bet, the forecast with a date and a number on it, sits in far shorter supply. An organization that grades itself on retrodiction will feel wiser every quarter and know nothing. The unfalsifiable theory isn't the one that fails the test; it's the one that arranges never to sit the exam.
The mapping
| Mathematics | Life |
|---|---|
| training folds | The life that taught you the belief — the colleagues, the exes, the childhood it was fitted to. |
| the held-out fold | The life the belief hasn't met yet: the next friend, the next job, the next decade. |
| in-sample fit | How convincing the theory feels from inside — graded by the very past that shaped it. |
| CV score | How the belief actually greets the new — the only grade that was ever about the world. |
| the journal | Predictions registered before their outcomes: a forecast without a date is a mood. |
| the revisionist | The mind that grades itself on retrodiction — a fresh perfect story for every month, none ever at risk. |
Where the metaphor tears
Cross-validation assumes the hidden fold is drawn from the same world as the training folds. Your next decade is not: the distribution drifts — you age, the culture turns, the people change. So even a perfectly honest CV score overstates what a belief has earned. Validation bounds your optimism; it cannot guarantee the future. In the instrument the truth never moves. In a life, it does.
A vow is not a prediction about your spouse; a faith held in exile is not an estimate. Some beliefs exist to constitute a life rather than predict one — they make the future rather than guess it. Grading those on held-out data is a category slip: scoring the foundation on how well it forecasts the house. Cross-validate your models of the world; be careful cross-validating your commitments to it.
Hiding data from yourself sounds free and is not. The real cost of the journal is living with registered misses — a written record of being wrong that no retelling can soften. The revisionist's method exists because it is anesthetic. That is why almost nobody keeps the journal, and why the few who do hold an advantage that looks, from the outside, like luck.