Cardinality problems are invisible until the bill or the query latency makes them obvious. A budget catches the runaway label before either happens.
A metric with a user ID or a request ID as a label dimension does not look dangerous when it is added. It looks dangerous three weeks later, when the number of distinct time series has grown past a million and every dashboard querying it has gotten noticeably slower to load.
1. Measure cardinality per metric before it becomes a problem
Helix tracks the distinct series count for every metric automatically. Query it directly rather than waiting to notice a slow dashboard, that is usually the first symptom, not the first cause.
from metrics
| summarize distinct_series = dcount(series_id) by metric.name
| sort by distinct_series desc
| limit 202. Set a budget per metric, not one number for the whole account
A single account-wide cardinality limit either blocks a legitimate high-cardinality metric or lets a runaway one through unchecked. Budgets scoped per metric name catch the actual offender.
helix cardinality set-budget \
--metric "checkout.request_duration" \
--max-series 5000 \
--on-exceed reject_new_series3. Find the label actually driving the growth
When a budget alert fires, the useful next step is not deleting the metric, it is identifying which single label dimension is responsible, so only that one gets dropped or bucketed.
from metrics
| where metric.name == "checkout.request_duration"
| summarize dcount(series_id) by label_key
| sort by dcount(series_id) descNine times out of ten this points at exactly one label, a user ID, a session token, a raw URL path with an embedded ID, that has no business being a metric dimension. Removing that one label, and moving the detail it carried into a trace attribute instead, usually resolves the entire budget breach without losing any real diagnostic value.
4. Review budgets quarterly, not just when they break
A metric that was safely under budget at launch can drift as a product adds features and label values multiply. A quarterly review of the top twenty metrics by series count catches slow drift before it becomes an incident.
- Query distinct series count per metric before a slow dashboard forces the question.
- Set budgets per metric name, not a single account-wide ceiling.
- When a budget breaks, find the offending label dimension before deleting anything.
- Review the top series-count metrics quarterly, cardinality drifts, it rarely spikes without warning.