A flaky network link between a collector and Helix causes retries. Retries cause duplicate spans. Here's how the ingest path keeps duplicates from doubling your trace counts.

OTLP exporters retry on ambiguous failures — a timeout doesn't tell you whether the server received and processed the batch or not. Retrying is the correct, safe default, but it means Helix's ingest path has to assume duplicate spans arrive regularly, not as a rare edge case.

How often this actually happens

Across our fleet, we measure retry-induced duplicate spans at roughly 0.3% of total span volume in steady state, spiking to 2-3% during network-degraded periods or collector restarts. At 40 million spans/day for a mid-size tenant, even the steady-state rate is 120,000 potential duplicate spans daily — enough to visibly skew count-based metrics if left unhandled.

The dedup key

Each span carries a deterministic identity: trace_id + span_id, which OTel guarantees is stable across retries of the same span (the exporter doesn't regenerate IDs on retry, it resends the same object). Ingest checks incoming span_ids against a per-shard bloom filter of recently-seen span_ids — a rolling 10-minute window, since duplicates from retries arrive within seconds to low minutes of the original in practice.

ingest:
  span_dedup:
    enabled: true
    window: 10m
    key: 'trace_id + span_id'
    on_duplicate: drop_and_count   # metric: helix.ingest.duplicate_spans_dropped

Why a bloom filter and not an exact set

An exact deduplication set for 10 minutes of span_ids at fleet scale would mean tracking potentially hundreds of millions of UUIDs in memory. The bloom filter, tuned to a 0.5% false-positive rate at this scale, costs a fraction of that memory — roughly 180MB per shard for our reference throughput — at the cost of very rarely dropping a legitimately unique span that happens to collide. We accept that tradeoff because a rare dropped unique span is less harmful than either unbounded memory growth or letting real duplicates through.

What this means for counts

Before dedup shipped, we saw customer-reported anomalies where `count()` aggregations over spans during a known network-flaky period ran 2-4% high compared to expected traffic — entirely attributable to retry duplicates. Post-dedup, that discrepancy dropped to under 0.1% in the same conditions.

  • Retry-induced duplicate spans run about 0.3% of volume in steady state, up to 2-3% during network degradation.
  • Dedup uses a rolling 10-minute bloom filter keyed on trace_id + span_id, costing ~180MB per shard.
  • Post-dedup, count anomalies during flaky-network periods dropped from 2-4% to under 0.1%.