Latency, traffic, errors, and saturation cover most of it — but we found a fifth signal worth dashboarding on every service we own.

The four golden signals get you most of the way to knowing whether a service is healthy. The fifth one we added — queue depth trend — caught the failures the other four missed until it was too late.

Why saturation alone wasn't enough

Saturation as usually defined (CPU, memory, disk) told us a box was busy, not that it was about to fall over. Our worst incidents were preceded by a slowly growing queue backlog for 20–40 minutes while CPU and memory looked completely normal — the work was piling up faster than workers could drain it, and none of the standard four signals showed it clearly.

Trend, not level, is the useful part

A queue depth of 500 means nothing on its own — it depends on throughput. What matters is the second derivative: is the backlog growing faster than it's draining? We alert on sustained positive slope, not on an absolute depth.

from metrics("queue.depth")
| where queue == "payments-worker"
| summarize depth = avg(value) by bin(_time, 1m)
| extend slope = depth - prev(depth)
| summarize sustained_growth = countif(slope > 0) over(last 10)
| where sustained_growth >= 8

Roll it out per service, not org-wide

Not every service has a meaningful fifth signal — a stateless read-only API doesn't have a queue to watch. We made the fifth signal a required field in the service catalog ("what is your saturation leading indicator, if any") rather than mandating queue depth everywhere, which forced teams to actually think about their own failure mode instead of copy-pasting a dashboard.

A batch-heavy data pipeline picked partition lag as its fifth signal instead of queue depth, and a connection-pooled API picked pool-wait time. None of those would have shown up if we'd mandated one metric name for every service — the value was in the question, not in the specific number.

  • Keep the four golden signals, but ask each service what its leading indicator is.
  • Alert on trend (sustained slope) rather than absolute level for backlog-style signals.
  • Make the fifth signal a required, service-specific field, not a copy-pasted dashboard.
  • Validate the fifth signal against a real past incident before trusting it.