Tail-based sampling needs the whole trace before it can decide. That means buffering, and buffering has a bill. Here's how we size it.

Head-based sampling decides at the first span. Tail-based sampling waits until a trace is complete — or times out — before deciding whether to keep it. That decision quality is worth a lot: we keep 100% of error traces and p99 outliers instead of a random 1%. But it means holding every open trace in memory until it closes.

Sizing the buffer

Each collector instance holds a bounded LRU of in-flight traces, keyed by trace_id, with spans appended as they arrive. At our default settings — a 30-second completion window and a 90KB average trace footprint for a 40-span request — a collector handling 8,000 traces/sec needs roughly 8,000 × 30 × 90KB, or about 21.6GB of buffer headroom at steady state, before any safety margin.

sampling:
  mode: tail_based
  decision_window: 30s
  buffer:
    max_traces_in_flight: 250000
    max_memory_mb: 24576
  policies:
    - name: keep_errors
      condition: 'status == "error"'
      sample_rate: 1.0
    - name: keep_slow
      condition: 'duration > p99_baseline'
      sample_rate: 1.0
    - name: baseline
      sample_rate: 0.02

What happens when the window closes early

Not every trace finishes cleanly — a lost span, a crashed sidecar, a client that never sends the closing segment. When the 30-second window expires with an incomplete trace, we make the sampling decision on whatever spans arrived rather than discarding the whole trace, and flag it `partial: true`. In steady state, about 0.6% of traces fall into this bucket; during a downstream outage that number can spike past 4% as retries and timeouts fragment spans across restarts.

The memory-vs-window tradeoff

Widening the decision window to catch slower traces (some fan-out requests legitimately take 45-60s) linearly increases buffer requirements. We landed on 30s as the default because it covers our p99.9 request duration across customers, and pushing to 60s would nearly double buffer memory for a marginal 0.3% improvement in trace completeness — not a good trade for most deployments, though we expose it as a tunable for latency-heavy workloads like batch APIs.

  • Tail-based sampling trades memory for decision quality: 100% error/outlier retention instead of blind random sampling.
  • Buffer sizing is a direct function of traces/sec × window × average trace footprint.
  • Incomplete traces at window expiry are sampled on partial data and flagged, not discarded.