semanticquant · collection · one event, every code
A Production Incident
Green dashboards, dead checkout — 2:07 a.m.
One event, four instruments. Each section reads the same outage in its own code; colour marks the destination site.
The page fires at 2:07 a.m. Checkout is timing out, and the dashboards are green. Yesterday's deploy shipped at five and ran clean for nine hours — until the nightly export began, and a service that answers in forty milliseconds started answering in four seconds. Clients timed out and retried; the retries tripled the load; the load lengthened the answers. By 2:30 three engineers are typing over each other in the incident channel. At 3:15 someone turns the retries off, and the system, its extra work gone, walks itself back to normal. The postmortem is at ten.
MatheMetaphor Tail latency in a queue without slack
The service was provisioned to the average, and averages are what queues punish. Queueing and slack gives the curve: as utilization approaches capacity, waiting time turns vertical — the distance between 85% busy and 96% busy is the distance between an ordinary night and this one, and the nightly export was the last eleven points. Slack reads as waste on every capacity review and as the decisive asset at 2 a.m.; both readings price the same idle machines. The green dashboards are a tail latency artifact: the median request was fine all night, and checkout dies in the tail — a page assembled from thirty calls samples the 99th percentile thirty times, so one slow answer in a hundred touches nearly every customer. The outage was visible for weeks in a p99 column nobody graphed.
CommuniCoupling A loop feeding on its own output
The mechanism is a vicious circle: slow answers cause timeouts, timeouts cause retries, retries triple the offered load, and the load slows the answers — each client running an individually reasonable policy that amplifies the deviation it was written to survive. The simulation would show the signature: load still climbing after the original disturbance has passed, because the loop no longer needs its trigger. Turning the retries off repairs nothing directly — it starves the loop, and the service walks home on its own. The channel at 2:30 runs the other model. Three people typing over each other is a noisy channel at capacity, and noise and redundancy says the remedy is never talking faster: an incident commander, readbacks, one thread per hypothesis are redundancy spent deliberately, so that the sentence "the retries are the load" survives the noise it has to cross.
SocioRhetoric What the channel's sentences were doing
"Mitigation is out, watching the graphs" has the grammar of the Report — assertive, keyed to reality — and pressure bends the key: the 2:40 update to the executive thread performs control as much as it describes it, and "we're fine" is the channel's likeliest specimen of belonging talk in a report's syntax. The morning meeting opens with a frame-set: "this postmortem is blameless" is a declaration fixing which moves are legal for the next hour — "my deploy did this" becomes a contribution of data, and "whose deploy was it" becomes an illegal move rather than the obvious question. That frame is what the engineer's 2:12 "sorry — that was my deploy" was waiting for: outside it, an Apology asking absolution; inside it, the timeline's first entry.
ExploRational Moves for before, during, and after
The document worth having tonight was never written: a spring premortem — "it is next year and the nightly batch took down checkout; write the history" — would have put the retry policy and the export on the same page while the fix was a config line. Inside the incident the working move is listening for the back-talk: the restart that changes nothing and the load that keeps climbing after mitigation are the system answering the engineers' moves, and the answer — load without new demand — is what points to retries at 3:15. The postmortem's method is genealogy: not who typed the deploy but by what sequence of reasonable decisions aggressive retries became the default — the policy dates to a 2019 outage where retrying saved the night — because a postmortem that stops at the deploy schedules the sequel.
The four readings disagree about what failed at 2:07: a queue run too close to capacity, a loop that outlived its cause, a channel whose reports had stopped reporting, or a question that was cheap in the spring and never asked.