Head-based sampling throws away the traces you'd actually want to see before it knows which ones those are. Tail-based sampling is out of beta today.

For eight months, tail-based sampling for traces lived behind a flag while we hammered on it with eleven design partners. Today it's on by default for every new Helix workspace, and available to flip on for every existing one.

The traces you need after an incident are almost never the ones a random 1-in-100 sampler kept.

The problem with deciding at the head

Classic sampling makes its keep/drop decision at the very first span, before the request has done anything interesting. It has no idea the downstream call is about to time out, or that this particular route is about to blow past your SLO. So you sample uniformly, keep 1-2% of everything, and discover during the retro that the exact request you need was one of the 98% you threw away.

How tail-based sampling decides after the fact

Helix now buffers spans for a configurable window (default 30 seconds) at the edge collector, waits for the trace to complete, and only then applies your keep policy. Errors, high latency, and specific routes can all be kept at 100% while the boring, fast, successful traffic gets sampled down hard. The policy is just HelixQL:

sampling policy "checkout-tail" {
  keep: status == "error" or duration > 800ms or route == "/checkout"
  window: 30s
  default_rate: 0.02
}

Everything that doesn't match falls through to default_rate, so you're not paying to store every healthy request while still catching the one in ten thousand that mattered.

What changed since the beta

Design partners pushed us hard on three things: the buffering window needed to be tunable per-service (a batch job's "slow" is a checkout API's "catastrophic"), the collector needed to survive a restart without dropping in-flight windows, and policy changes needed to apply without redeploying agents. All three shipped. One partner cut their trace storage bill 61% while keeping 100% of their error traces, up from roughly 4% under uniform sampling.

Rolling it out without a big-bang cutover

We were careful not to force every workspace onto tail-based sampling overnight. New workspaces get it by default; existing ones see a banner with a one-click preview that runs the new policy against the last 24 hours of real traffic before anything changes, so nobody is surprised by what gets kept or dropped. Roughly two-thirds of eligible workspaces migrated within the first month, almost entirely self-serve, without a single support ticket asking where a specific trace went.

  • Tail-based sampling policies as HelixQL, versioned like everything else
  • Per-service buffering windows, default 30s
  • Zero-downtime policy updates, no agent redeploy
  • Typical storage reduction of 40-60% with error/slow-path traces kept at 100%