We cut pages per on-call week from 34 to 6 in one quarter without hiding real problems. Here is the plan, week by week.

Nobody fixes alert fatigue with one afternoon of alert tuning. It took us a structured 90-day plan and a weekly review to get pager volume down without losing signal.

Weeks 1–2: inventory before you touch anything

We pulled every alert that fired in the last quarter and ranked it by fire count and by whether the responder took any action beyond acknowledging it. Alerts that fired often and never led to action were the first target — not because they were wrong, but because they had trained everyone to ignore pages.

from alert_history
| where fired_at > ago(90d)
| summarize fires = count(), actioned = countif(resolution != "no_action") by alert_name
| extend action_rate = actioned * 1.0 / fires
| where fires > 20 and action_rate < 0.1
| order by fires desc

Weeks 3–8: fix the alert, not the symptom

For each noisy alert we asked one question: is this a symptom-based alert (users are affected right now) or a cause-based alert (something looks off but might self-heal)? Cause-based alerts got demoted to a dashboard or a ticket, never a page. This was the single biggest volume reduction — roughly 60% of our pages were cause-based alerts wearing a symptom-based badge.

Weeks 9–12: add burn-rate alerts to replace static thresholds

Static thresholds like "CPU over 80% for 5 minutes" page on noise. We replaced the worst offenders with multi-window burn-rate alerts tied to actual SLOs, which only fire when the error budget is genuinely at risk, not when a metric wobbles.

What we tracked to know it was working

We didn't just watch total page count drop — a naive team could hit zero pages by deleting everything, which would be a disaster dressed up as success. Alongside volume we tracked mean time to acknowledge and post-incident survey scores from the rotation each week, and both improved in step with the volume drop, which is what told us we'd removed noise and not signal.

  • Inventory every alert by fire count and action rate before changing anything.
  • Separate symptom-based alerts (page) from cause-based alerts (dashboard/ticket).
  • Replace static thresholds with SLO-based burn-rate alerts where it makes sense.
  • Track acknowledge time and rotation sentiment alongside volume, not volume alone.
  • Re-run the inventory query monthly — noise creeps back.