Shard by trace_id and a single trace lives on one node, but a service-wide query fans out everywhere. Shard by service and it's the reverse. Helix does neither purely.

Every distributed trace store faces this choice at the storage layer: shard spans by trace_id, so an entire trace always lives together, or shard by service, so a query for "all checkout-api spans" hits one place. Each choice makes the other query pattern expensive. Helix uses a hybrid.

The pure trace_id-sharding cost

Sharding purely by trace_id (hash of trace_id mod shard count) makes single-trace lookups perfect — one shard, one lookup. But it scatters a service's spans uniformly across every shard, so a query like "p99 duration for checkout-api over the last hour" has to fan out to all N shards even though it never needs to reconstruct a full trace. On a 64-shard cluster, that's 64x the query fan-out for a query pattern that's arguably more common than single-trace lookups.

The pure service-sharding cost

The reverse holds too: shard by service, and service-wide aggregation is fast and local, but a single trace spans multiple services by definition — a checkout request touches checkout-api, payments-api, and inventory-api spans, which now live on three different shards, meaning single-trace reconstruction always fans out.

Helix's hybrid: shard by service, index by trace_id

from traces
| where service == 'checkout-api' and duration > 1s   # single-shard, fast
| summarize p99(duration) by route

from traces
| where trace_id == 'a93f7c2e...'                       # multi-shard fan-out
| project service, span.name, duration

We primarily shard by service, since service-scoped aggregation is the dominant query pattern in dashboards and alerting — roughly 78% of production trace queries in our telemetry are service-scoped, not trace-scoped. Single-trace lookups fan out across the (typically small, 3-6 service) set of shards involved, located via the bloom-filter segment index described elsewhere, rather than scattering to all shards.

Measured cost of the hybrid

Service-scoped aggregation queries average 90ms on a 64-shard cluster, versus an estimated 310ms if trace_id sharding forced full fan-out for the same query. Single-trace lookups average 140ms under service sharding — slower than the theoretical best case of pure trace_id sharding (roughly 40ms) but that's a cost we accept for the majority query pattern winning by 3.4x.

  • Pure trace_id sharding optimizes single-trace lookups but forces full cluster fan-out for service-scoped queries.
  • Pure service sharding is the reverse; Helix shards by service since 78% of production queries are service-scoped.
  • Single-trace lookups under service sharding fan out only to the small set of shards actually involved, not the whole cluster.