Every point needs to survive a crash before it's acknowledged. Group commit is what keeps that durability guarantee from tanking throughput.

Acknowledging a write before it's durable is a data-loss bug waiting for a bad day. Fsyncing on every individual point is durable but slow — a single fsync on our reference NVMe hardware costs around 0.4-0.8ms, which would cap single-threaded ingest at roughly 1,500 points/sec if done per point. Helix's WAL uses group commit to get both.

How group commit works

Writes from many concurrent ingest connections accumulate into a shared per-shard buffer. A dedicated WAL writer thread flushes and fsyncs that buffer on whichever comes first: a 10ms timer or a 4MB buffer fill. Every writer waiting on that fsync gets acknowledged together, in one syscall, regardless of how many individual writes were batched.

wal:
  group_commit:
    max_wait_ms: 10
    max_buffer_mb: 4
  fsync_mode: full   # options: full, none (dangerous), os_buffered

What this buys in practice

On a shard receiving 12,000 points/sec across 200 concurrent connections, group commit means roughly one fsync every 10ms instead of 12,000 — a 99.2% reduction in fsync calls, at the cost of adding up to 10ms of latency to the slowest write in each batch. Measured p50 durability latency including WAL fsync: 6.8ms. p99: 14.2ms, mostly batches that landed right at the buffer-fill boundary.

Why the WAL is per-shard, not global

A single global WAL would serialize every write in the cluster through one fsync stream, an obvious bottleneck. Per-shard WALs let fsync throughput scale roughly linearly with shard count, since each shard's writer thread and disk queue are independent. The tradeoff is that a crash recovery has to replay N independent WALs instead of one, but that only matters at restart, not on the hot path.

Recovery cost

On restart after an unclean shutdown, WAL replay for a shard with a typical 4MB of unflushed-to-chunk-storage data takes under 400ms — we cap unreplayed WAL size specifically to keep recovery time bounded, since a shard that can't come back online within a second or two starts affecting cluster-wide availability during a rolling restart.

  • Group commit batches concurrent writes into one fsync per 10ms or 4MB, whichever comes first.
  • Measured p50 durability latency: 6.8ms; p99: 14.2ms on standard NVMe hardware.
  • Per-shard WALs scale fsync throughput linearly with shard count, at the cost of N-way recovery replay.