Every point Helix stores crosses the same six stages in under 12ms end to end. Here's what happens between the OTLP receiver and the durable write.

A metric point or span arrives as an OTLP protobuf, and by the time it's durable, it has passed through parsing, validation, cardinality checks, batching, WAL append, and index update — typically in under 12ms at p50. Here's the path, stage by stage.

Stage 1-3: receive, validate, budget-check

The OTLP receiver decodes protobuf into an internal row format, validates required fields (timestamp, metric name, at least one label), and runs the fingerprint hash against the tenant's cardinality budget described elsewhere. These three stages together average 0.8ms per batch of 500 points.

Stage 4: batching before the WAL

Rather than fsync per point, points are accumulated into per-shard batches on a 50ms timer or a 4,000-point count threshold, whichever comes first. This is the single biggest lever on ingest latency: a smaller batch window lowers latency but increases fsync frequency and CPU overhead from syscalls.

from helix_internal.ingest
| where stage in ('receive', 'validate', 'wal_append', 'index_update')
| summarize p50(duration_ms), p99(duration_ms) by stage
| where p99 > 20

Stage 5-6: WAL append and async index update

The batch is appended to a per-shard write-ahead log with a group fsync, averaging 3.1ms at p50 and 9.4ms at p99 on our standard NVMe-backed nodes. Once durable, an acknowledgment returns to the client, and index update (chunk assignment, dictionary interning) happens asynchronously off the critical path — so a point is durable and safe from loss before it's queryable, with a typical 200-400ms gap between "written" and "visible to queries."

Where the 12ms actually goes

On a representative production shard: 0.8ms validate, 0.3ms budget-check, up to 50ms batching wait (but averaging 18ms since batches fill early under load), 3.1ms WAL fsync, rest in queueing. Under load spikes, batching wait is the first thing to grow, which is by design — it's the cheapest place to absorb backpressure before the WAL itself becomes the bottleneck.

  • End-to-end p50 durable-write latency is under 12ms; p99 sits around 34ms on standard hardware.
  • Batching before WAL append is the primary latency/throughput lever, tuned by a 50ms timer or 4,000-point threshold.
  • Index update happens async after durability, creating a 200-400ms window before new points are queryable.