A bad escalation policy either pages a whole team for a minor issue or leaves a major incident unstaffed while primary sleeps through it. We rebuilt ours around severity.

Twice in one month our escalation policy failed in opposite directions: once it paged four people for a blip, once it let a genuine outage sit unstaffed for twenty minutes because primary's phone was on silent.

Escalate on missed acknowledgment, tied to severity

Low-severity alerts wait a full 10 minutes before escalating past primary, giving room for someone mid-task to notice and acknowledge. High-severity alerts tied to SLO burn escalate after 3 minutes and notify secondary in parallel rather than sequentially, because those extra minutes of an unstaffed critical incident cost more than the risk of a slightly redundant page.

from pages
| where acknowledged_at == null and _time < ago(3m)
| where severity == "critical"
| project page_id, primary_responder, service, escalate_to = secondary_responder

Don't escalate to a whole team by default

Our old policy's fallback for any missed acknowledgment was paging the entire team channel, which trained people to assume someone else would answer and made acknowledgment slower, not faster, over time. We replaced that with a strict, ordered chain — primary, then secondary, then the engineering manager — and reserved whole-team paging for a small, explicit list of the highest-severity scenarios.

Test escalation chains the way you test failover

We schedule a quarterly synthetic escalation test — a fake page that's designed to go unacknowledged by primary on purpose — to confirm secondary actually receives it within the expected window. Two of these tests caught a misconfigured phone number and an expired push-notification token that would otherwise have failed silently during a real incident.

Keep the chain current when people change roles

An escalation chain is only as good as the roster behind it, and rosters drift constantly — someone changes teams, someone goes on leave, someone's phone number changes and nobody thinks to update the paging tool. We tied the escalation chain to the same identity system that manages team membership, so a role change updates the chain automatically instead of relying on someone remembering a second system exists.

  • Tie escalation timing to severity — critical alerts escalate faster and in parallel.
  • Replace default whole-team paging with a strict, ordered escalation chain.
  • Reserve whole-team paging for a small, explicit set of highest-severity scenarios.
  • Run a quarterly synthetic escalation test to catch silent failures in the chain.
  • Tie the chain to your identity/team-membership system so role changes update it automatically.