A runbook is a snapshot of a system that keeps changing. Here is how we catch the moment a documented fix stops matching reality.

We found a runbook telling engineers to SSH into a box that had been decommissioned four months earlier. Nobody had noticed because nobody had needed that runbook since.

Link runbooks to the services they describe

Every runbook now references a service ID in the service catalog. When that service's infrastructure changes in a way we track — a host migration, a queue replacement, a major dependency swap — the linked runbooks get automatically flagged as needing review instead of silently drifting.

Verify by running, not by reading

A reviewer re-reading a runbook and nodding along is not verification — they're pattern-matching that it sounds plausible. Real verification means running the confirmation query and, where safe, the first remediation step against a staging environment on a schedule.

from runbook_verifications
| where verified_at < ago(90d) or verified_at == null
| where linked_service in (active_services)
| project runbook_id, owner, last_verified_days_ago = datetime_diff('day', now(), verified_at)
| order by last_verified_days_ago desc

Archive aggressively

A stale runbook is worse than no runbook, because it gets followed with false confidence during an incident. We archive anything unverified for 180 days rather than letting it linger — an archived runbook that turns out to still be needed gets restored and re-verified in minutes; a rotted one followed blindly costs a lot more.

Assign verification to whoever changed the system

Instead of routing every flagged runbook back to its original author, who may have left the team or the company by the time it drifts, we route the re-verification task to whoever made the infrastructure change that triggered the flag. They already have the context on what's different, and it spreads runbook ownership across the team naturally instead of concentrating it on whoever happened to write the first draft two years ago.

  • Link every runbook to a service ID so infrastructure changes trigger a review flag.
  • Verify by actually running the confirmation query and safe steps, not by re-reading.
  • Track days-since-verified as a first-class metric, reviewed monthly.
  • Route re-verification to whoever made the triggering change, not always the original author.
  • Archive anything unverified past 180 days rather than letting it rot in place.