Two different metrics on the same pod share most of their label set. A per-metric dictionary wastes that overlap; a shared vocabulary captures it.

Dictionary encoding already cuts label-set storage substantially, as covered elsewhere, but a naive per-metric dictionary still misses an obvious win: `service=checkout-api, pod=checkout-7f9d` appears identically across a dozen different metric names on the same pod. A shared, cross-metric vocabulary captures that overlap.

Splitting label sets into components

Instead of dictionary-encoding a full label set as one opaque string, Helix decomposes it into individual key=value tokens, and interns each token independently into a shard-wide token dictionary. A label set then becomes a short array of token IDs rather than one string ID, letting `service=checkout-api` share a single dictionary entry across every metric and every pod that carries it.

# token dictionary (shared across all metrics on the shard)
1024: 'service=checkout-api'
1025: 'region=us-east-1'
1026: 'pod=checkout-7f9d-x92k1'

# a label set becomes an array of token IDs
label_set: [1024, 1025, 1026]   # 12 bytes vs ~68 bytes as a raw string, vs 4 bytes for a whole-set ID

Where the extra win comes from

The whole-set dictionary from our earlier design already got label sets down to 4 bytes per reference, which sounds like it beats the 12-byte token-array approach on paper. The real win is elsewhere: with a whole-set dictionary, `region=us-east-1` existing inside 50 different metrics' worth of label sets meant 50 different whole-set dictionary entries, each storing that substring redundantly. Token-level sharing collapses that redundancy at the dictionary level itself, not just at the reference level.

Measured impact

On a representative shard carrying 180 distinct metric names across 12,000 pods, the whole-set dictionary held about 940,000 entries with significant internal string redundancy, totaling roughly 61MB compressed. The token-vocabulary redesign reduced the underlying dictionary to about 38,000 unique tokens (service names, pod names, region values, etc.) at just 4.1MB compressed — a 93% reduction in dictionary size, even though individual label-set references grew slightly.

The lookup cost tradeoff

Resolving a full label set now means N token lookups instead of one whole-set lookup, where N is typically 4-8 for a realistic label set. Measured average resolution cost went from about 40ns (single lookup) to roughly 140ns (multiple token lookups) — a real but small cost, easily absorbed given how rarely label sets need full string resolution versus being compared as token-ID arrays directly.

  • Label sets are decomposed into individual key=value tokens, shared across all metrics on a shard.
  • Measured dictionary size reduction: 93% (61MB to 4.1MB) on a representative 12,000-pod shard.
  • Full label-set resolution cost rose from ~40ns to ~140ns, a worthwhile tradeoff given the storage win.