Step 4 said restart the ingest workers. Step 4 had been wrong for three months, and nobody had run it since the architecture changed underneath it.

Step 4 said restart the ingest workers. Step 4 had been wrong for three months, and nobody had run it since the architecture changed underneath it.

Following the Book Exactly

The on-call engineer that night did everything right. Ingest lag was climbing, the runbook for "ingest lag high" said to restart the ingest-worker deployment, and she ran the documented command. Lag didn't budge. She restarted it again, thinking maybe it hadn't taken effect. Twenty minutes gone, still climbing, and now she was starting to doubt her own diagnosis rather than the runbook, because the runbook is supposed to be the source of truth.

What the Architecture Actually Looked Like

Three months earlier, we'd split the monolithic ingest-worker into two independently scaled services: a fast-path writer and a batch reconciler. The runbook had never been updated. Restarting "ingest-worker" now restarted a service that had been reduced to a thin compatibility shim doing almost nothing, while the actual bottleneck — the batch reconciler — sat untouched:

from metrics
| where name == "ingest.lag_seconds"
| where service == "ingest-reconciler"
| summarize max(value) by bin(5m)
| where value > 300

That query, run against the right service, would have shown the real culprit in under a minute. Instead it took a second engineer joining the bridge, unfamiliar enough with the "current" architecture to ask "wait, do we still have a reconciler service?" to break the loop.

Runbooks Rot Faster Than You Think

We instituted a rule that any architecture change touching a service named in an active runbook requires updating that runbook in the same pull request — enforced by a lightweight check that greps runbook service references against a manifest of current service names. It's caught four stale references since we added it, none of which had caused an incident yet, which is exactly the point: we'd rather find them boring and early than at 2am under pressure.

A runbook that's wrong is worse than no runbook, because it burns the confidence people need to deviate from it when it stops matching reality.
  • Runbooks decay the moment the architecture they describe changes, and nobody notices until they're needed.
  • Tie runbook updates to the pull requests that change the systems they describe, not to a quarterly review.
  • A lightweight automated check (service names still exist) catches a surprising fraction of rot for free.
  • When a runbook step doesn't work the first time, treat that as a signal to question the runbook, not just retry the step.