Pages that cry wolf train teams to ignore them. A framework for alerts that earn the interrupt.
A pager that fires for everything trains people to look at nothing. Alert fatigue is not a discipline failure on the responder's part — it is a design failure in the alerts themselves.
Every alert must be actionable
If a page arrives and the only response is to acknowledge it and go back to sleep, it should never have been a page. The test is simple: name the action a human takes when this fires. If you cannot, it is a dashboard signal, not an interrupt.
Alert on symptoms, not causes
Page on what the user feels — elevated error rate, latency past the SLO — not on every CPU spike or restarted pod that may or may not matter. Cause-based alerts multiply with your infrastructure; symptom-based alerts stay roughly constant because your users' expectations do.
Route by ownership, suppress by context
- Ownership routing sends each page to the team that can actually fix it, so nobody triages alerts they cannot act on.
- Deploy-window suppression mutes the noise that a known, in-progress change is expected to produce.
- Grouping collapses a storm of related alerts into one incident with one owner.
Review the noisy ones weekly
Track which alerts fire most and resolve without action, and delete or tune them on a schedule. An alerting system that is never pruned only grows louder — and a loud system is one your team has already learned to ignore.