We almost migrated everything to burn-rate alerting and dropped every static threshold. A near-miss taught us why both still earn a place.

Burn-rate alerting felt like a strict upgrade over static thresholds, so we started migrating everything. A near-miss with a slow memory leak stopped us halfway through.

What burn-rate alerts are bad at

Burn-rate alerts are tied to an SLO window, which means they're blind to problems that haven't yet produced enough bad events to move the ratio — a memory leak climbing steadily toward an OOM kill produces zero SLI-visible badness until the moment it crosses the line and takes the service down all at once. A static threshold on memory usage would have caught it hours earlier.

from metrics("process.memory.rss")
| where service == "search-indexer"
| summarize mem = avg(value) by bin(_time, 5m)
| where mem > 0.85 * container_limit

What static thresholds are bad at

Static thresholds don't know what's actually SLO-relevant, so they either page on things that don't matter to users or need constant hand-tuning as normal usage patterns shift with traffic growth. We keep them for leading-indicator, cause-based signals — memory, disk, queue depth — and use burn rate for anything measuring actual user-facing impact.

The dividing line we settled on

If the signal directly represents user experience, it becomes a burn-rate alert against an SLO. If it's a leading indicator that predicts future user impact before it happens, it stays a static or trend-based threshold, usually routed to a non-paging channel unless it's climbing fast. Neither approach replaces the other; they cover different failure shapes.

Document the pairing per service, not just in principle

Every service's alert catalog now lists its burn-rate alerts and its leading-indicator thresholds side by side, with a one-line note on what failure each one is meant to catch. That forced us to notice two services that had burn-rate coverage but no leading indicators at all — the exact gap that caused the near-miss in the first place — and fix them before they repeated it.

  • Use burn-rate alerts for anything that directly measures user-facing impact.
  • Keep static or trend thresholds for leading indicators like memory and queue depth.
  • Don't migrate everything to burn rate — it's blind to problems before they hit the SLI.
  • Route leading-indicator alerts to a non-paging channel unless they're climbing fast.
  • Document both alert types per service so gaps in the pairing are easy to spot.