Ingest latency is invisible until it is not — until the logs you need are still thirty seconds behind the incident. Here is how we closed the gap.
Batch on bytes, not count
Fixed-count batches starve under low traffic and choke under bursts. Sizing batches by bytes kept the pipeline smooth across both.
Backpressure beats buffering
Unbounded buffers turn a slow consumer into an out-of-memory crash. Explicit backpressure made the slow path slow instead of fatal.
Measure at the p95, not the mean
Averages hide the incidents. The mean ingest latency looked healthy the whole time we were paging; the p95 was the number that actually tracked how far behind reality our logs were. We rebuilt the dashboard around tail latency and the improvement finally became visible.
Where the 60% actually came from
- Byte-sized batching smoothed throughput across quiet nights and traffic spikes alike.
- Explicit backpressure turned an out-of-memory crash into a graceful, recoverable slowdown.
- Zero-copy parsing on the hot path shaved the per-line CPU cost that dominated at volume.
None of the three was heroic on its own. Stacked, they took the p95 from just over a second to comfortably under 400ms — and kept it there under load.