The version of tail-based sampling we shipped to GA looks meaningfully different from what we designed on a whiteboard. Here's what customers changed.

Design docs are always wrong in some way you can't predict until real workloads hit the feature. Tail-based sampling's beta ran for six months across eleven design partners, and three specific pieces of feedback changed the shape of what shipped to general availability.

Fixed 30-second windows didn't fit batch workloads

Our original design used one global buffering window. A checkout API partner was thrilled with 30 seconds. A data-pipeline partner running batch jobs that legitimately take four minutes end-to-end found their "slow" traces getting cut off mid-buffer and evaluated incomplete. We shipped per-service window overrides in response, not a single global setting.

Policy changes needed to be safe to test

Early beta users were understandably nervous about editing a sampling policy in production — get the keep condition wrong and you silently lose the traces you need during the next incident. We added a dry-run mode that reports what a policy change would have kept over the last 24 hours of real traffic, before it goes live:

hx sampling dry-run --policy checkout-tail.yaml --window 24h

Policy would have kept: 4.8% of traces (up from 2.1%)
Errors captured: 100% (unchanged)
Estimated storage delta: +$340/mo

The collector needed to survive being restarted

The most operationally serious bug we found in beta: a rolling deploy of the collector itself could drop in-flight buffering windows, silently losing exactly the traces the feature exists to protect. We rebuilt the buffer to checkpoint to local disk and replay on restart before this shipped to GA — a partner's chaos-engineering team found this for us, and we're grateful they did.

What we're watching next

GA doesn't mean we consider the feature finished. The dry-run and per-service window overrides both came from beta feedback, and we expect GA usage at ten times the scale to surface a different class of edge case entirely, probably around extremely bursty traffic patterns that beta's steadier design-partner workloads didn't stress heavily. We're keeping the same rapid-iteration cadence on this feature post-GA that we ran during beta, rather than treating the GA label as a reason to slow down.

  • Per-service buffering windows, not a single global setting
  • Dry-run mode to preview a policy change against real traffic before going live
  • Collector-level checkpointing so a restart can't silently drop buffered traces
  • All three changes came directly from design partner production usage, not internal design review