A mobile client buffering telemetry offline for six hours will eventually flush it all at once. Helix's watermarking decides how long to wait before calling a window closed.

Not every point arrives close to its own timestamp. Retry queues, offline-first mobile SDKs, and batched exporters can deliver data minutes or hours after it was generated. A rollup or sampling decision that closes its window too early silently drops that data; one that never closes never finishes computing.

How the watermark advances

Each ingestion shard tracks a watermark — the timestamp before which we're willing to say "no more data is coming." It advances based on the observed lateness distribution of recent traffic, not a fixed clock offset: if the shard has seen 99.5% of points for a given minute arrive within 45 seconds historically, the watermark trails real time by roughly that margin, recalculated every few minutes.

ingest:
  watermark:
    strategy: percentile_lateness
    target_percentile: 99.5
    min_delay: 30s
    max_delay: 10m

What happens to data that misses the watermark

Points arriving after the watermark has passed their timestamp aren't dropped — they're written to their correct time bucket in raw storage, but any rollup or tail-sampling decision already made for that window isn't retroactively recomputed by default. This is a deliberate tradeoff: recomputing closed rollups for late data would mean every downstream consumer (dashboards, alerts, exemplars) needs to handle values changing after the fact, which is worse for on-call trust than a rollup being very slightly under-counted.

The opt-in correction path

For workloads where correctness matters more than stability — billing-adjacent metrics, for instance — pipelines can enable `max_delay: 10m` with retroactive rollup correction, at the cost of holding rollup state open 10 minutes longer and a small compute overhead (roughly 4% more CPU on the compaction path) for periodic re-aggregation checks.

What we've measured in practice

Across our fleet, the default 99.5th-percentile watermark strategy captures 99.5% of points inside their correct window while adding a median 38-second delay before a 1-minute rollup is considered final — a delay dashboards absorb invisibly since they're rendering rollups a few minutes old anyway.

  • Watermarks advance based on observed lateness percentiles, not a fixed offset.
  • Late data past the watermark is stored correctly but doesn't retroactively reopen closed rollups by default.
  • An opt-in correction mode exists for billing-sensitive pipelines, at roughly 4% extra compaction CPU.