signal & secrecy · metaphor 68 of 100
How relevant is one thing to another — really? Not correlated, not associated, not "linked in studies": how much does knowing X actually shrink your uncertainty about Y? That question has an exact answer, in bits — and it catches relationships correlation cannot see.
"Does the weather affect his mood?" "Does the interview predict the hire?" "Does this symptom tell us anything?" Everyday life runs on relevance judgments, and nearly all of them are made by vibe. We call things "very related" when they arrive together dramatically, and "unrelated" when the connection is inconvenient to trace. What we almost never do is ask the question in a form that has an answer: before I knew X, how uncertain was I about Y — and how uncertain am I now?
Mutual information is relevance audited. It is the number of yes/no questions about Y that knowing X saves you, on average. Zero means genuinely nothing — a rarer and more precious verdict than it sounds. And because it weighs every pattern at once, not just straight lines, it cannot be fooled by curves, rings, or relationships that swing both ways. Build a world below and read its relevance off the meter.
the joint table · paint how often each pair of facts occurs together
weather · X
the meter
Entropy H(Y) is how much you don't know about Y, counted in the fairest currency there is: the average number of yes/no questions a perfect questioner would need to pin Y down. Four equally likely moods cost two questions; a mood that's nearly always "fine" costs almost none. Then someone tells you X — the weather, the interview score, the symptom — and your remaining uncertainty is H(Y|X), averaged over everything X might turn out to be.
Mutual information is the difference: the questions you no longer have to ask. That subtraction is what your intuition performs, badly, every time you decide whether something "matters." The instrument performs it exactly. Getting I to its maximum requires a diagonal — each X value must commit to a Y value. And getting it to exactly zero requires every column to be the same distribution — a discipline almost no real pair of variables achieves, which is why "truly irrelevant" is a stronger claim than it feels.
the wager
Here is the same quantity as a game. The machine draws real pairs from the table above. Two ideal guessers bet on Y: one blind, one shown X first. Each pays the fair price of its surprise — the questions its answer would have cost. Run a few hundred draws and watch the two averages settle: the gap between them is the mutual information, felt as a standing advantage.
the guessing game · draws from the current joint table
what to try
Switch the skin to symptom × disease, keep the noisy channel preset, and run draw 500. The blind guesser's average settles at H(Y); the informed one's below it. That standing gap is what the symptom is worth — most symptoms are worth less than the fear they generate.
Clear the grid and paint a clean diagonal: I equals H(Y) exactly — X tells you everything. Then flatten it so every column looks alike: I hits 0.00. Notice how contrived the second world is. Genuine irrelevance is a claim about every column at once.
Below, load the ring. Correlation reads r ≈ 0 — "nothing here." Mutual information reads about 0.76 bits — "knowing x pins y to two narrow bands." Then load the straight line and read the third chip for the opposite lesson.
the blind spot
Pearson's r answers one narrow question: how well does a straight line fit? A ring, a V, a population where the effect runs one way for some people and the other way for the rest — all of these can average out to r ≈ 0 while X remains massively informative about Y. "No correlation" and "no relationship" are different sentences, and much folk statistics — and some published statistics — quietly swaps them.
There is a converse discipline too. A high r certifies that a line fits — not that the fine detail of X is earning its keep. On the straight-line preset, compare I(X;Y) with the information carried by a one-bit summary of X ("high or low?"): the coarse question already captures most of the relevance. Impressive correlations often reduce to one honest bit wearing a lab coat.
the correlation trap · r vs. I on the same cloud
the leak
Almost nothing you know arrived firsthand. Between you and the weather stands a forecast; between you and the candidate, an interviewer's paragraph; between you and the organization, a dashboard. Information theory has a hard law about such chains — the data-processing inequality: if all you see is a summary of X, the summary's relevance to Y can never exceed X's own. Processing can compress, reorganize, clarify. It cannot add back a bit that the summarizer dropped.
This turns some soft intuitions rigid. Gossip about gossip is a lossy channel run twice. Minutes of a meeting are an executive's chosen partition of what happened. Every dashboard is someone's decision about which distinctions deserve to survive. Choosing your informants and your instruments is choosing how many bits reach you — which is why "I heard it firsthand" is a number, and the panel below computes it for the world you painted above.
the chain · X → summary → you (uses the joint table)
the mapping
| Mathematics | Life |
|---|---|
| H(Y) | How uncertain you are before — the honest size of the open question, in yes/no questions. |
| H(Y|X) | How uncertain you remain after the fact arrives — what the evidence leaves unfinished. |
| I(X;Y) | The relevance of X to Y: the questions its answer dissolves, counted, not felt. |
| I = 0 | Genuinely uninformative — every version of X leaves Y exactly as unknown. Rarer than it feels. |
| the ring dataset | The relationships correlation calls "nothing": curved, forked, two-way — invisible to a straight line, plain in bits. |
| data processing | What dies between the source and you — each retelling, summary, and dashboard can only shed bits, never mint them. |
where the metaphor tears
Estimating I from data is treacherous: with few samples and many bins, noise masquerades as structure, and the plug-in estimate is biased upward — everything looks a little relevant. That is why the scatter panel discloses its bin count and sample size and applies a bias correction. Anyone quoting mutual information without those numbers is quoting their optimism.
I(X;Y) is perfectly symmetric: the weather is exactly as informative about his mood as his mood is about the weather. It measures how much two variables know about each other, and stays silent about who is moving whom, or whether a third thing moves both. For direction and intervention you need a different instrument entirely — see do-calculus.
One bit about a biopsy and one bit about tomorrow's drizzle are, to the mathematics, the same bit. Information theory counts uncertainty resolved; it does not know what the answer costs you, or what you would trade to have it. A life spent maximizing bits per hour would be exquisitely informed about nothing that matters. The weighting of questions by consequence is the part it leaves — correctly — to you.