I own eleven alert rules. I decided to grade my own homework and found four of them had no business waking anyone up.

I own eleven alert rules. I decided to grade my own homework and found four of them had no business waking anyone up.

Nobody Audits Their Own Alerts

It's easy to review someone else's noisy rule in a retro and suggest tuning it. It's harder to look at the alerts you personally wrote and admit half of them were set defensively — "just in case" thresholds from a moment of anxiety after a past incident, never revisited once the anxiety faded. I own the alerting for our query-planner service, and I finally sat down and pulled the full firing history for everything I'm the owner of.

Grading My Own Rules

For each rule I checked how often it fired, and of those firings, how many led to an actual code or config change versus a shrug:

from incidents
| where alert_owner == "priya.raman"
| summarize fired = count(), acted_on = countif(resolution == "changed_something") by alert_rule
| project alert_rule, fired, acted_on, rate = acted_on * 100.0 / fired
| sort by fired desc

Two rules had a 100% action rate on small sample sizes — genuinely good, high-signal alerts. Four others had fired a combined 43 times over the year with a 0% action rate. Every single firing was closed as "transient, recovered on its own" or "within acceptable range, will monitor." Those four were pure cost: forty-three separate moments where someone, often me, got interrupted for nothing.

What I Did About It

I deleted one rule outright — a query-planner cache hit ratio threshold that had never once correlated with a user-visible problem in two years of data. I widened the threshold on two others to match what the data showed was actually the boundary of "fine" versus "not fine." And I converted the last one from a page into a dashboard annotation, since it was useful context during other incidents but never on its own worth an interruption. Eleven rules became eight, and I haven't missed the three that left.

The alert you wrote out of anxiety six months ago is not obligated to keep paging you today.
  • Audit your own alert rules with the same rigor you'd apply to someone else's — ownership bias is real.
  • Track "fired vs. led to a change" per rule; a 0% action rate over dozens of firings is a strong deletion signal.
  • Not every useful signal needs to be a page — some belong as passive context on other incidents instead.
  • Deleting a rule is a legitimate outcome of tuning, not a failure to tune it properly.