We paged three engineers at 4am for a threshold that was correct in every way except that it should never have existed.

We paged three engineers at 4am for a threshold that was correct in every way except that it should never have existed.

A Perfectly Correct, Perfectly Useless Alert

The rule fired exactly as written: error rate on auth-service crossed 2% for five minutes. Nobody disputes the math. What nobody had checked in fourteen months was whether 2% still meant anything. Auth had grown from 400 requests per second to 6,000. At the old volume, 2% errors was eight failed logins a second — worth waking someone up. At the new volume it was 120 failed logins a second, but it turned out those were almost entirely bots retrying expired tokens, a pattern that had been growing gradually for months without ever crossing the threshold in a way anyone noticed until that one Tuesday night.

Finding the Alerts That Fire on Autopilot

We ran an audit across every alert rule that had fired more than five times in the last quarter, ranked by how often the on-call engineer closed it within two minutes with no action taken — our proxy for "this alert wastes people's sleep":

from incidents
| where status == "resolved" and resolution == "no_action"
| where time_to_resolve < 2m
| summarize count() by alert_rule
| sort by count() desc
| limit 15

Fourteen rules showed up. Nine of them, like the auth-service one, were static thresholds set when the service was a fraction of its current size and never revisited. We rewrote them as relative alerts — error rate compared to the trailing 7-day baseline for that hour of day — instead of absolute numbers.

The Uncomfortable Part

The audit surfaced something harder to fix than the thresholds: three of those fourteen rules had fired more than sixty times each over the quarter, and every single time the response was "looked fine, closing." Sixty pages, zero incidents. That's not noise, that's a design defect we'd been quietly tolerating because each individual instance felt too small to escalate. It took the aggregate view to make the cost visible.

An alert that's correct on the day you write it can become noise a year later without a single line of code changing.
  • Static thresholds decay as traffic and baselines shift — review them on a schedule, not just after an incident.
  • Track "paged, no action taken" as its own metric; it's the clearest signal of alert debt you have.
  • Relative baselines (vs. trailing 7-day, same hour) age better than fixed numbers for traffic that grows.
  • If a rule pages more than it helps, retiring it is a bigger win than tuning it.