Four pages in six hours. Three were nothing. The fourth was the reason the other three almost got ignored.

Four pages in six hours. Three were nothing. The fourth was the reason the other three almost got ignored.

8:14pm — Disk Usage Warning

First page: disk usage on a logging sidecar crossed 85%. I've seen this exact page a dozen times; log rotation lags behind write volume during peak hours and catches up within twenty minutes. I acknowledged it, checked that the trend was flattening, and closed it in ninety seconds without further action.

9:40pm — Elevated Latency, Recovered

Second page: p99 latency on an internal admin API crossed threshold for exactly the five-minute window required to fire, then recovered on its own before I'd even opened the laptop. Low-traffic service, single-digit requests per minute at that hour, so one slow request can swing p99 dramatically without meaning anything. Closed, no action.

10:55pm — The One That Mattered

Third page looked, on the surface, identical to the second: elevated latency, different service. My hand was already moving to acknowledge-and-dismiss out of pattern-matching fatigue when I made myself run one query before closing it:

from traces
| where service == "payments-svc"
| where duration > 3s
| summarize count() by bin(1m)
| sort by bin(1m) desc
| limit 10

Unlike the admin API, this was a high-traffic service and the count wasn't a single outlier request — it was climbing steadily, minute over minute. That's the tell that separates noise from a real fire: volume and trend, not just crossing a threshold once. This one turned out to be a connection pool exhaustion building toward a full outage, caught about twelve minutes before it would have started failing payments outright.

The Habit That Almost Failed Me

Two legitimate false alarms in a row had built a reflex to dismiss the third without looking closely, and that reflex is exactly how real incidents get missed. The fix isn't "look closely at everything always" — that's not sustainable at 11pm on a Friday. It's a cheap discriminator: check count/trend, not just threshold-crossing, before dismissing anything on a high-traffic service specifically, since that's where a real trend is statistically meaningful in a way a single slow request on a quiet service isn't.

  • Pattern-fatigue from consecutive false alarms is a real failure mode, not just an annoyance.
  • On high-traffic services, a climbing trend over minutes is a stronger signal than a single threshold crossing.
  • On low-traffic services, single-request outliers can swing p99 without meaning anything — check volume before trusting the signal.
  • Give yourself one cheap, fast check before dismissing anything, even the fourth page of the night.