We estimated multi-region ingest at one quarter. It took four. The gap between those numbers is the actual lesson here.

Multi-region ingest sounds, from the outside, like a networking problem: accept data closer to where it's generated, forward it to wherever it's stored. We scoped it that way too, at one quarter. It took closer to a year, and almost none of the delay was the networking part.

The part we scoped correctly

Standing up regional ingest points in three additional regions and routing traffic to the nearest one was, genuinely, about as much work as we estimated — a few weeks of infrastructure work per region.

The part we didn't: ordering guarantees across regions

HelixQL's tail-based sampling and cross-signal joins both assume a consistent, mostly-ordered view of events within a buffering window. Once ingest happens in three regions simultaneously and gets merged into one logical workspace, "mostly ordered" stopped being true, and sampling policies started making keep/drop decisions on incomplete windows because a chunk of a trace's spans hadn't arrived from another region yet.

from ingest_lag
| where signal == "traces"
| summarize p99(cross_region_lag_ms) by source_region, dest_region
// worst observed pair, pre-fix: p99 lag 4,200ms -> sampling windows too short

What actually fixed it

We extended the tail-sampling buffer window dynamically based on observed cross-region lag rather than a fixed default, and added a region-aware trace assembler that waits for a trace-completeness signal instead of a flat timer when spans are known to be arriving from more than one region. That work, not the ingest routing, was the actual year.

What we'd tell ourselves a year ago

Any feature touching a system with an implicit ordering assumption — and tail-based sampling has one baked deep into its design — deserves a distributed-systems-shaped estimate the moment "multi-region" enters the conversation, not a networking-shaped one.

What shipped alongside it

Because we were already deep in the trace assembler's internals, we used the same completeness-signal work to fix a longstanding, unrelated bug where a single very slow downstream span could hold a trace's sampling decision open far longer than its buffering window intended, even within a single region. That fix shipped as a side effect of the multi-region project and quietly improved sampling accuracy for every customer, not just the ones using multiple regions.

  • Multi-region ingest across three additional regions, routed to nearest point
  • Dynamic sampling buffer windows based on observed cross-region lag
  • Region-aware trace assembler replacing a flat completeness timer
  • The real lesson: ordering assumptions, not networking, drove the timeline