The retro wasn't about the outage. It was about the eleven engineers who got paged for a problem that needed exactly two of them.

The retro wasn't about the outage. It was about the eleven engineers who got paged for a problem that needed exactly two of them.

The Outage Was Fine. The Paging Wasn't.

The incident itself was resolved in eighteen minutes — a config rollback for a bad feature flag default. What stood out in the retro was the escalation log: our "page everyone on the service's team plus two adjacent teams" policy, written years earlier for a different, more ambiguous class of incident, had fired for something whose owner was identifiable within the first ninety seconds.

Measuring the Actual Cost

We pulled every paged engineer across the incident's lifecycle and cross-referenced who actually took an action versus who just acknowledged and waited:

from incidents
| where id == "INC-5502"
| join actions on incident_id
| summarize actions = count() by engineer
| sort by actions desc

Eleven engineers paged. Two took any action at all. Nine people were woken up, or pulled out of dinner, or interrupted mid-focus-block, to acknowledge a page and then do nothing, because the escalation policy couldn't distinguish "we don't know who owns this" from "we know exactly who owns this and it's these two people."

Rewriting the Policy Around Confidence

We split escalation into two tiers based on whether the triggering alert carries a confident service attribution. High-confidence alerts — ones tied to a specific service's own SLO breach — now page only that service's primary and secondary. Low-confidence, cross-cutting symptoms — like a platform-wide latency spike with no clear single owner — still trigger the wider net, because that ambiguity is exactly what the broad policy was designed for. The nine unnecessary pages weren't a one-off; they were a rule mismatched to its own worst case, applied uniformly to every case.

A broad escalation policy is right for the incidents that are genuinely ambiguous, and wrong for everything else. Most incidents aren't ambiguous.
  • Audit not just whether an escalation policy resolved the incident, but how many paged people actually acted.
  • Tier escalation by attribution confidence, not by severity alone — a clearly-owned Sev-1 doesn't need eleven people.
  • A policy written for the hardest, most ambiguous case will over-page every easy case that follows the same rule.
  • Unnecessary pages are a real cost even when the incident itself resolves quickly and cleanly.