Every golden signal was healthy. Customers were still stuck. The gap between our dashboard and their experience taught us we were measuring the wrong thing.

Every golden signal was healthy. Customers were still stuck. The gap between our dashboard and their experience taught us we were measuring the wrong thing.

The Dashboard Said Everything Was Fine

Support tickets started trickling in about search results not updating after a product edit. Our search-service dashboard showed the four golden signals — latency, traffic, errors, saturation — all green. Latency was actually better than average that day. Error rate was near zero. If I'd trusted the dashboard alone, I'd have closed this as "can't reproduce."

The Fifth Signal We Weren't Tracking

What golden signals don't capture is correctness of eventual state, only the health of the request path getting there. We had to build a new query on the fly to check something we'd never instrumented: the gap between when a product update was written and when it became visible in search:

from traces
| where service == "search-indexer"
| where span.name == "index.apply"
| summarize p95(duration), p99(duration) by bin(15m)
| where p99(duration) > 10m

Indexing lag, a metric that existed but had no alert and no dashboard tile, had crept from a normal 20 seconds to over eleven minutes. Requests were fast, errors were zero, saturation looked fine — because the indexer wasn't failing or overloaded, it was just working through a backlog caused by a batch reindex job someone kicked off manually and forgot was still running, silently competing for the same worker pool as real-time updates.

Adding a Fifth Signal

The four golden signals are the right default, but they assume "the request succeeded" is the same as "the system did the right thing." For anything with an async or eventually-consistent component — indexing, replication, cache invalidation — we now treat freshness lag as a first-class fifth signal with its own SLO and its own alert, separate from the synchronous request path. It would have caught this in minutes instead of the ninety it took support escalations to reach us.

A green dashboard means your instrumented signals are healthy. It says nothing about the signals you never thought to instrument.
  • The four golden signals measure request-path health, not correctness of eventual state.
  • Any async or eventually-consistent component needs a freshness/lag metric as a first-class signal, not an afterthought.
  • Manual one-off jobs (reindexes, backfills) need the same visibility and lifecycle as scheduled ones.
  • When the dashboard and the support queue disagree, trust that there's a signal you haven't built yet.