A single slow consumer group fell behind by six million messages over a weekend, and our end-to-end freshness SLO paid for it all week.

Between Saturday and Monday, the metrics-aggregation consumer group fell 6.1 million messages behind its Kafka topic. Nobody noticed until Monday morning, because the alert we had was on consumer errors, and this consumer wasn't erroring — it was just too slow.

Timeline

Saturday 02:00 — a scheduled partition rebalance, triggered by an unrelated autoscaling event, briefly pauses all consumers in the metrics-aggregation group for processing. 02:04 — consumers resume, but one of the six new consumer instances is scheduled onto a node with degraded disk I/O (a known-bad node that hadn't yet been cordoned). 02:04–Monday 08:00 — that one consumer processes messages at roughly a third of normal throughput, and because partition assignment isn't dynamically rebalanced for throughput, its partitions simply fall behind continuously. Monday 08:15 — on-call notices metrics ingest freshness SLO (p95 end-to-end delay < 60s) has been breached since Saturday morning, discovered via the weekly SLO report rather than a live page. 08:40 — degraded node identified and cordoned; consumer group scales up temporarily to accelerate catch-up. Monday 14:00 — backlog fully drained.

Root cause

We had alerting on consumer group errors and on total lag in message count, but our lag alert threshold (1 million messages) was tuned for a burst scenario, not a slow continuous drift — by the time it crossed 1 million on Saturday afternoon, the alert did fire, but it routed to a low-priority Slack channel instead of paging, based on an old severity mapping.

The fix

The query that made the degraded node obvious in hindsight:

from metrics
| where metric.name == "kafka.consumer.lag" and group == "metrics-aggregation"
| summarize lag=max(value) by partition, bin(time, 15m)
| where lag > 500000

One partition's lag climbed steadily while the other five stayed flat — a clear single-consumer bottleneck, not a group-wide problem. We fixed the immediate issue by cordoning the bad node and added automatic per-partition lag-rate alerting (lag growing, not just lag being high) so a slow-drift case pages the same as a burst.

What changed

The 1-million-message lag alert now pages, full stop — no more routing based on a stale severity table. We also added node health (disk I/O latency) as a factor in Kafka consumer scheduling constraints, so consumers avoid landing on already-degraded nodes.

  • Alert on lag growth rate per partition, not just total lag — it catches single-consumer bottlenecks.
  • Audit alert routing/severity mappings regularly; a correctly-firing alert that goes to the wrong channel is a silent outage.
  • Exclude known-degraded nodes from consumer scheduling until they're fully remediated.
  • Don't let a weekly report be the first place an SLO breach is discovered.