Every metrics store enforces a cardinality limit eventually, usually by falling over. We decided to expose ours on purpose, before it becomes an outage.

Cardinality explosions are the single most common cause of a metrics backend degrading, and historically the failure mode is silent until it isn't: a new label with a user ID or a UUID in it multiplies series count by orders of magnitude, and the first anyone hears about it is a query timeout during an incident. We decided customers deserved to see the limit coming, not just hit it.

Budgets instead of hard walls

Every metric namespace now gets a cardinality budget, visible before it's exceeded, with a live series-count meter:

from cardinality_usage
| where metric == "http_request_duration"
| summarize series=dcount(label_set) by bin(1h)
| order by bin desc

At 80% of budget, Helix identifies the specific label driving growth — almost always one offending label, not a diffuse problem — and surfaces it directly:

from cardinality_usage
| where metric == "http_request_duration"
| top_label_contributor
// result: label "request_id" contributes 94% of series growth

Why we didn't just raise the limits

Bigger infrastructure could absorb higher cardinality, but it wouldn't fix the underlying problem: a runaway label is almost always a mistake, not a legitimate need for that many series. Surfacing it early turns a future outage into a five-minute fix — remove the label, or move it to a log field where high cardinality is cheap and expected.

The warning that mattered most in practice

The single highest-value case turned out to be catching a bad deploy before it fully rolled out: a new build that accidentally logged a per-request UUID as a metric label shows up in the cardinality-growth warning within minutes, well before the full fleet has rolled the change and well before the metrics backend would otherwise have started slowing down under the weight of millions of new series. Two customers have told us this alone caught a deploy that would have been a multi-hour incident under their old, unattributed metrics setup.

  • Per-namespace cardinality budgets with a live series-count meter
  • Automatic identification of the single label driving growth
  • Warnings before a limit is hit, not a failure after
  • Guidance toward moving high-cardinality data to logs where it belongs