A runbook written for the person you are today is useless to the person you will be at 3 a.m. — tired, paged, and half-awake. Write for that person instead.
The best runbook is not the most thorough one. It is the one a groggy engineer can follow correctly at 3 a.m. without making the incident worse.
Write for the worst moment, not the best
You write runbooks calm and caffeinated. They get read panicked and half-asleep. Every ambiguity you leave becomes a decision someone has to make under pressure, and pressure is where mistakes live.
The shape of a good runbook
- Symptom first — start with what the pager said, so people can match the alert to the page in seconds.
- Copy-pasteable commands — no "adjust the config accordingly." Give the exact command, with placeholders clearly marked.
- A blast-radius note — say what each step affects before it is run, not after.
- An escalation path — who to wake, and after how long, when the runbook runs out.
Link the query, do not describe it
Instead of telling someone to "check ingest lag," embed the query and the link:
metrics
| where __name__ == "ingest_lag_seconds"
| stats max(value) by pipeline
| sort by value descTest them like code
A runbook that has never been run is a hypothesis. We rehearse ours during game days, and every step that trips someone up gets rewritten. A runbook is a living document or it is a lie you tell during onboarding.