Not every alert deserves to interrupt someone's night. We rebuilt our routing rules around actual urgency instead of defaulting everything to a page.

We were paging our on-call engineer at 2 a.m. for issues that could safely wait until business hours. That's not diligence, it's a design failure in the routing layer.

Classify by real urgency, not by source

Every alert used to route to a page by default unless someone remembered to configure otherwise, which meant urgency was decided by omission. We flipped the default: every new alert starts as a non-paging notification, and only alerts explicitly tied to user-facing impact or SLO burn get promoted to paging, reviewed at creation time.

from alert_definitions
| where routing == "page" and linked_slo == null
| project alert_name, owner, created_at
| order by created_at desc

Time-of-day aware routing for genuinely borderline alerts

A handful of alerts are urgent during business hours but tolerable overnight — a batch job running slow, for instance, matters far less at 3 a.m. than at 9 a.m. when people are waiting on its output. Those route to a page during business hours and to a digest queue overnight, reviewed first thing by whoever starts their day next.

Measure interrupted sleep as its own metric

We now track pages that fire between 11 p.m. and 6 a.m. local time to the responder as a distinct number from total page volume. Overall pages can go down while overnight pages stay flat if you're not watching this specifically — and overnight pages are the ones that actually drive burnout and attrition.

Respect local time, not server time

Our first pass at this used UTC cutoffs across a rotation that spans three time zones, which meant "overnight" for one responder was mid-afternoon for another and the routing rules were silently wrong for two-thirds of the team. Routing now resolves against each responder's actual local time at the moment of the page, which sounds obvious in hindsight but took a real bug report to notice.

  • Default new alerts to non-paging; require an explicit reason to promote to a page.
  • Use time-of-day aware routing for alerts that are urgent by day but tolerable overnight.
  • Track overnight pages as a distinct metric from total page volume.
  • Resolve overnight windows against each responder's local time, not server time.
  • Review alerts that page without a linked SLO — that's usually a routing mistake.