A runbook that only makes sense to the person who wrote it isn't a runbook, it's a note to self. Here is the format we standardized on.

We audited 40 runbooks and found the same failure mode in 30 of them: they were written by an expert, for an expert, and useless to whoever was actually on call at 3 a.m.

Lead with the query that confirms the symptom

Every runbook now opens with a copy-pasteable HelixQL query that confirms whether the alert's story is true, before any remediation step. This alone cut false-start remediations by a lot — engineers stopped restarting services based on a hunch and started confirming first.

from logs
| where service == "payments-worker" and level == "error"
| where message contains "connection pool exhausted"
| summarize count() by bin(_time, 1m)
| where count_ > 5

Write steps as commands, not descriptions

"Check if the queue is backing up" is not a step. "Run this query, if the value is above 5000, go to step 4" is a step. We rewrote every runbook so each line is either a command to run, a query to check, or an explicit branch — no prose that requires interpretation under stress.

Put an owner and a last-verified date at the top

Unowned runbooks rot silently — a service gets refactored and the remediation steps quietly stop working, and nobody finds out until an incident. Every runbook now has an owner and a last-verified date; anything unverified for 90 days gets flagged in the weekly reliability review and someone has to either re-verify it or archive it.

Test the format on someone who didn't write it

Before a runbook is considered done, we hand it to an engineer who has never touched the underlying service and time how long it takes them to reach the correct remediation, purely from the document. If that person gets stuck or has to ask a Slack question to proceed, the runbook fails review and goes back for another pass — the author's fluency with the system doesn't count as evidence the page will work at 3 a.m.

  • Open with a query that confirms the symptom before any remediation.
  • Write steps as commands and explicit branches, not prose.
  • Require an owner and a last-verified date on every runbook.
  • Test every runbook on someone unfamiliar with the service before publishing.
  • Archive or re-verify anything untouched for 90 days.