Our alert thresholds were tuned for daytime traffic. At 2am, the same absolute error count that should have paged someone was statistically invisible.

A background worker pool deadlocked at 02:03 on February 9th. It wasn't a subtle failure — the job queue backed up from a normal depth of 200 to over 40,000 — but nobody was paged until a customer in a different timezone opened a ticket at 02:50.

Timeline

02:03:00 — a deploy of the export-worker service introduces a mutex ordering bug that deadlocks under a specific combination of two job types running concurrently. 02:04:00 — job queue depth starts climbing steadily. 02:03–02:50 — our alert on "error rate > 5% of job volume" never fires, because the deadlocked workers aren't erroring, they're just silently not processing. 02:50:12 — a customer opens a support ticket about a stuck export. 02:54:00 — support escalates to on-call. 03:01:00 — responder identifies the deadlock via thread-state inspection and a HelixQL query on job age. 03:10:00 — workers restarted, backlog begins draining. 03:35:00 — queue back to normal depth.

Root cause

We had exactly one alert on this queue: error rate. We had no alert on queue depth or on job age — the two signals that would have caught a "processing has silently stopped" failure mode rather than a "processing is failing loudly" one. Both are incidents; only one paged.

The fix

The query that should have been a standing alert:

from logs
| where service == "export-worker" and message == "job.dequeued"
| summarize oldest_job_age=max(now() - job.enqueued_at) by bin(time, 1m)
| where oldest_job_age > 5m

Run retroactively, oldest_job_age crossed 5 minutes at 02:06 — 44 minutes before the customer ticket. We added this as a burn-rate style alert on job age and a second one on queue-depth growth rate, both independent of error rate.

What changed

We audited every queue-backed service for the same gap and found four others with only an error-rate alert and no staleness or depth alert. All four now have age-based alerting. We also stopped assuming low absolute traffic (2am) makes an alert threshold safe to skip — the fix was a second, orthogonal signal, not a lower threshold on the same one.

  • Every queue needs both an error-rate alert and a staleness/age alert — they catch different failure modes.
  • Silent stalls don't show up in error-rate metrics; audit for this blind spot on every async pipeline.
  • Don't rely on absolute volume thresholds that behave differently at low-traffic hours.
  • A support ticket that arrives before a page is itself a finding — track time-to-page as its own metric.