the second hundred · metaphor 214
You threw yourself into a new city, a new love, a new self — and only later noticed how much of the old you had gone faint. Not betrayed, exactly. Just written over, without anyone deciding to.
Learning something new and keeping something old are, at a certain depth, the same act pulling in opposite directions. To absorb a new life you have to let your habits, reflexes and defaults change — and those very defaults are what carried the old life. Stay too rigid and the new thing never takes; stay too fluid and the old thing washes away. There is no dial marked “learn without forgetting.” There is only a trade, and you are always somewhere on it.
Neural networks hit this wall so hard it has a name: catastrophic forgetting. Train one on a task, then train it on a second, and the first can vanish almost completely — not degraded, erased, its weights repurposed for the new job. Beneath it is the stability–plasticity dilemma: the standing, unavoidable price of a system that must both hold the past and take on the present with one shared substrate.
The idea
A network learns by nudging thousands of shared weights until its output matches the training examples. Teach it the old life and the weights settle into a configuration that gets that task right. Now teach it the new life. Gradient descent has no memory of why the weights were where they were — it only knows the new errors, and it moves whatever weights reduce them. Many of those are the very weights the old task depended on. Nothing forbids their being overwritten, so they are.
The result is catastrophic forgetting: not a gentle decay but a near-total collapse of the first task while the second is mastered. The knowledge wasn't stored in a protected box; it was smeared across the same parameters the new task now claims. This is the stability–plasticity dilemma in its harshest form — plasticity, the capacity to change, is exactly what erases the past; stability, the capacity to hold, is exactly what blocks the new.
You can buy back some of the old life, but never for free. Rehearse the old examples while learning the new; or anchor the important weights so the new task must route around them. Crank that anchor up and the old life survives — but the new one can no longer land, because the network is now too rigid to move. The knob doesn't abolish the trade. It only lets you choose where on it to stand.
What to try
Leave the price at fully plastic. Teach the old life: the left panel sharpens into a clean split — the network has learned it, and its accuracy climbs near 100%. Now teach the new life and keep your eyes on the left panel, not the right. As the right panel (the new task) sharpens, the left one dissolves: its clean boundary smears, its accuracy falls, often all the way back to a coin-flip. You will watch a competence get quietly written over in a few seconds, by nothing but the pursuit of the new.
Now reset and raise the standing price. At a middling anchor, both panels hold a compromise — the old life is mostly kept, the new life mostly learned, neither perfect. Push the anchor to rigid and the old life survives almost intact, but the new task can barely form: the network is too stiff to change. There is no setting that gives you both, fully. That absence — not any single reading — is the instrument's real result.
The mapping
A person, like the network, runs their old life and their new one on one nervous system — the same reflexes, the same defaults, the same limited attention. Immersion in the new — the new job, city, partner, language, self — is genuine learning, and it works the same brutal way: it moves whatever it must to fit the present, including the grooves the old life ran in. The friend you stopped calling, the skill you let lapse, the version of you that knew a different set of streets by heart — often nobody chose to lose them. They were the weights the new life needed.
The dilemma explains why the two obvious cures both fail. Cling hard to the old self (crank up stability) and the new life never really takes — you're present but unchanged, unable to learn the place you've moved to. Dissolve completely into the new (full plasticity) and you look up to find the old life gone. The wise middle is rehearsal: deliberately revisiting the old — the calls, the practice, the return trips — so the shared substrate is reminded it still has to serve two masters. Not because the past is sacred, but because a system that overwrites without rehearsal keeps only its latest chapter.
Read as life lessons
Nothing decided to erase the old life; the machinery that learns the new simply reuses the same parts. Loss by immersion isn't disloyalty — it's the default behaviour of a shared substrate.
Every method that protects the past taxes the future: anchoring old weights, replaying old data, growing new capacity. You can move along the trade, but you cannot step off it.
What keeps an old competence alive is revisiting it while you learn the new — the calls, the practice, the trips home. Memory you never touch is memory the next chapter is free to claim.
In the wild
An entire field exists to stop it: elastic weight consolidation anchors important weights, replay buffers rehearse old data, and modular nets grow new capacity — all ways to hold the past without freezing the future.
Adapt a large model to a narrow new task and it can lose broad old abilities — “alignment tax” and skill regression. Practitioners mix in old data precisely to rehearse against catastrophic forgetting.
Biology fights the same dilemma with complementary systems — a fast, plastic hippocampus and a slow, stable cortex — and replays the day during sleep. Rehearsal, it turns out, is very old engineering.
The mapping, exactly
| Network | Life |
|---|---|
| the shared weights | One nervous system — the reflexes, defaults and attention that every part of your life must share. |
| training on task B | Immersing yourself in the new life — the move, the job, the love, the reinvention. |
| catastrophic forgetting | The old life going faint not by choice but as a by-product of learning the new. |
| plasticity | Your capacity to change — the very thing that lets the new take and the old wash away. |
| stability / anchoring | Clinging to who you were — protective of the past, but it blocks the new from landing. |
| rehearsal / replay | Deliberately revisiting the old — calls, practice, trips home — so the substrate keeps serving it. |
Where the metaphor tears
Brains have specialized systems — a fast-learning hippocampus, slow-consolidating cortex, sleep replay — that soften the dilemma in ways a plain network can't. People forget, but rarely as cleanly or catastrophically as this toy does. The metaphor captures a real pressure, then overstates how total the erasure usually is.
Not all forgetting is loss. Shedding an old identity, a bad habit, a grief you've outgrown is often exactly the healthy work of a new life. The network treats every old task as worth keeping; you don't have to. The dilemma tells you there's a cost — it does not tell you the old life was worth its price.
The starkest forgetting here comes from a small network with fixed size. A larger brain, or one that grows new structure for the new task, can hold both with far less interference. The dilemma is real but scale-dependent — “you must forget to grow” is truer of a crammed system than of one with room to spare.