Every label you add to a metric multiplies the number of stored time series. Left unmanaged, one well-meaning user_id tag can quietly 100x your bill overnight.
Why cardinality sneaks up on you
The trouble with cardinality is that it compounds. A metric with three labels — service, region, status_code — looks harmless until someone appends customer. Now the series count is the product of every distinct combination, and the graph of your storage cost bends sharply upward.
Set a budget, not a ban
Teams that fight cardinality with a blanket "no new labels" policy end up with metrics nobody can slice. A budget is friendlier: each service gets an allowance of active series, and adding a high-cardinality label means retiring something else.
- Reserve high-cardinality identifiers (user, request, trace id) for logs and traces, never metrics.
- Bucket continuous values — latency into histogram buckets, not one series per millisecond.
- Review the top ten metrics by series count every sprint; they drift.
Find your worst offenders in HelixQL
You can ask the store directly which metrics are eating the budget before finance does:
metrics
| where __name__ =~ "http_.*"
| stats series = count_distinct(series_id) by __name__
| sort series desc
| limit 10Make the budget visible
The single most effective change we made was putting each team's series count on the same dashboard as their error rate. Cardinality stops being an abstract platform concern the moment it sits next to a metric the team already cares about.