We were paging our on-call engineer eleven times a night. A quarter later it was three, and none of the fixes involved raising a threshold and hoping.

Eleven pages a night is not an alerting configuration, it is a resignation letter waiting to be written. Here is how we got to three without pretending the problems went away.

Count before you cut

We started by categorizing a month of pages. The result was uncomfortable and clarifying: most pages were not incidents. They were duplicates, non-actionable warnings, and the same flapping alert firing on a timer.

  • 40% duplicates — one root cause paging through five symptoms.
  • 25% non-actionable — "disk 80% full" with days of runway and no runbook.
  • 15% flapping — an alert crossing its threshold every few minutes.

The three fixes

Each category had a specific remedy. Deduplication grouped related alerts into one incident. Non-actionable warnings became tickets, not pages. Flapping got hysteresis — fire high, clear low — so a metric hovering at the line stopped ringing the bell.

alert "disk pressure"
| when disk_used > 0.9 for 15m
| clear when disk_used < 0.8
| route to ticket   # not a page

The number that mattered

We tracked pages per on-call shift and put it on a leadership dashboard. Making the toil visible to the people who set priorities did more than any single technical fix. Reliability work gets funded when the cost of not doing it is on a chart.

What we protected

The goal was never zero pages — a service that never pages is a service nobody is watching. The goal was that every page meant something. Three real pages beat eleven where you have stopped reading them.