A sustained ingest spike can outpace background compaction. Helix has to decide who slows down first, and it's a deliberately ordered list.

Under normal load, ingestion and compaction run at a comfortable pace relative to each other. Under a sustained spike — a customer's traffic doubling, or a large batch backfill — ingestion can outpace compaction's ability to keep segment counts under control. Something has to give, and Helix has a defined order for what gives first.

The signal that triggers backpressure

Each shard tracks its uncompacted segment count as the primary backpressure signal, rather than raw ingest rate, since segment count is what actually determines query cost degradation described elsewhere. Crossing a first threshold (default 20 uncompacted L0 segments) triggers the first response tier; a second threshold (40 segments) triggers a harder one.

Tier 1: reprioritize, don't reject

backpressure:
  tier1_threshold: 20
  tier1_action: boost_compaction_priority   # compaction io_priority: below_normal -> normal
  tier2_threshold: 40
  tier2_action: shed_low_priority_writes    # drop debug-level logs, sampled metrics first
  tier3_threshold: 80
  tier3_action: apply_ingest_backpressure   # 429 responses with retry-after

Tier 2 and 3: shedding and pushback

If boosting compaction priority alone doesn't stop segment count from climbing — typically because the spike is simply too large for available compaction throughput — tier 2 sheds the lowest-value writes first: debug-level logs and already-sampled-down metrics, which customers configure as explicitly droppable under load. Tier 3, reached only in sustained severe spikes (we've hit it a handful of times fleet-wide in the past year), applies real ingest backpressure: 429 responses with a retry-after header, pushing the problem back to client-side buffering rather than letting the shard fall further behind.

Why segment count and not queue depth

We considered using WAL queue depth as the backpressure signal instead, since it's more directly tied to write latency. We rejected it because queue depth is noisy on short timescales and reacts to momentary bursts that resolve themselves within seconds. Segment count is a slower-moving, more stable signal that better reflects genuine sustained pressure rather than normal traffic jitter — we measured roughly 4x fewer false-positive backpressure triggers using segment count over a 30-day comparison window. In the 14 months this system has been live, tier 1 activates a few times a week fleet-wide, invisible to customers beyond a brief compaction priority bump. Tier 2 activates roughly monthly, usually during a large customer's traffic event, and is visible only if a team explicitly opted debug logs into the droppable category. Tier 3 has fired 3 times, always during a genuine multi-minute traffic anomaly, and resolved within 90 seconds of the spike subsiding each time.

  • Backpressure escalates in three tiers, gated on uncompacted segment count, not raw ingest rate or queue depth.
  • Tier 1 reprioritizes compaction; tier 2 sheds low-value writes; tier 3 applies real 429 backpressure to clients.
  • Segment count as the trigger signal produced roughly 4x fewer false-positive activations than queue depth in testing.