Head-based sampling decides before a request finishes. Tail-based sampling waits, and keeps the traces that turned out to matter. Here is how to enable it for one service.

Head-based sampling makes the keep-or-drop decision at the start of a request, before anyone knows whether it will fail or run slow. That means a flat ten percent sample rate keeps ten percent of the failures too, exactly the traces most worth keeping at one hundred percent. Tail-based sampling fixes this by deciding at the end instead.

1. Route traces through the sampling collector

Tail-based sampling needs to see a complete trace before deciding, which means buffering spans for a window rather than exporting them immediately. Point the service at a collector configured for tail sampling instead of exporting directly to Helix.

# otel-collector-config.yaml
processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 500 }
      - name: sample-the-rest
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

2. Order the policies deliberately

Tail sampling policies evaluate in order, and the first matching policy decides the outcome for that trace. Put the always-keep rules, errors and high latency, ahead of the probabilistic catch-all, or a percentage rule evaluated first can discard a trace that a later error rule would have kept.

3. Watch the decision-wait window against real latency

The collector cannot make a keep-or-drop decision until it has seen the trace finish, or until the wait window expires, whichever comes first. Set decision_wait comfortably above the slowest normal request for the service, or genuinely slow-but-fine requests will get cut off mid-trace and evaluated as incomplete.

from traces
| where service.name == "checkout-api"
| summarize p99(duration)

If p99 duration is close to the configured decision_wait, raise the window before enabling the policy in production.

4. Confirm the sample rate landed where expected

After a day of traffic, check the actual kept percentage against the intended one, tail sampling configuration is easy to get subtly wrong, and the only reliable way to know it worked is to measure it.

from traces
| summarize errors_kept = countif(status_code >= 500) * 1.0 / countif(status_code >= 500 or sampled == false)
  • Keep one hundred percent of errors and slow requests, sample the rest at a lower rate.
  • Order policies with always-keep rules first, the probabilistic catch-all last.
  • Set decision_wait above the p99 duration for the service, not below it.
  • Measure the actual kept percentage after enabling, do not assume the config did what it says.