Finding one trace_id among billions of spans shouldn't require touching every segment. A per-segment bloom filter turns most lookups into a negative in microseconds.

A single-trace lookup — "show me trace a93f7c2e..." — is the most common query shape in Helix's tracing UI, and also the easiest to get catastrophically wrong at scale. With 40 million spans a day and 90 days of retention, that's up to 3.6 billion spans a lookup could theoretically touch.

Why a global index doesn't work here

A single global trace_id index sounds appealing, but trace_ids are effectively random UUIDs — there's no locality to exploit, and a B-tree or hash index over billions of entries, kept mutable as new segments arrive and old ones expire, becomes a write-amplification and compaction headache of its own.

The bloom filter approach

Instead, every immutable trace segment (roughly 15 minutes of spans, around 45,000 spans per segment at our reference load) gets a bloom filter built once at segment-close time, sized for a 0.1% false-positive rate at expected cardinality. Looking up a trace_id means checking the bloom filter for every segment in the retention window first — a check that costs about 180 nanoseconds per segment.

from traces
| where trace_id == 'a93f7c2e19b8...'
| where range == 90d
# planner checks ~8,640 segment bloom filters (15-min segments over 90d)
# before opening a single segment file

The math that makes this work

Checking 8,640 bloom filters at 180ns each costs about 1.6ms total — negligible. At a 0.1% false-positive rate, roughly 8-9 of those 8,640 segments will falsely report a possible match and need to be opened and checked directly, each costing around 4ms to scan its trace_id dictionary. Total worst-case cost for a 90-day trace lookup: roughly 1.6ms of bloom checks plus 35-40ms of false-positive verification — well under 50ms for a query that could otherwise mean opening thousands of segment files.

Sizing the false-positive rate

We tuned the 0.1% target by trading bloom filter memory against false-positive verification cost: a looser 1% rate shrinks filter memory by roughly 40% but multiplies false-positive segment opens tenfold, pushing worst-case lookup latency past 200ms. At our current 0.1% setting, bloom filters across a 90-day retention window for a mid-size tenant cost about 340MB of memory total — cheap compared to the alternative.

  • Every 15-minute trace segment gets an immutable bloom filter at close time, tuned to 0.1% false-positive rate.
  • A 90-day trace_id lookup costs roughly 1.6ms of bloom checks plus 35-40ms of false-positive verification.
  • Bloom filter memory for a mid-size tenant's 90-day retention: about 340MB total.