A liveness probe with too-tight thresholds started killing healthy pods under normal GC pauses, and Kubernetes dutifully restarted them into the same trap.

For roughly three hours on June 17th, the query-service fleet was stuck in a restart loop. Individual pods were healthy — the liveness probe just didn't think so, and Kubernetes kept killing and replacing them faster than they could stabilize.

Timeline

07:00:00 — a routine traffic increase (start of business hours) pushes query-service's JVM into more frequent, slightly longer GC pauses, a normal daily pattern. 07:02:00 — the liveness probe, configured with a 1-second timeout and no tolerance for a pause longer than that, starts failing during GC pauses that briefly block the health endpoint. 07:03:00 — after 3 consecutive failures (a 15-second window), Kubernetes kills the pod and starts a replacement. 07:03–10:15 — the replacement pod, under the same load and same JVM warm-up-induced GC pattern, fails its own liveness probe within minutes of starting, restarting the cycle. Across the fleet of roughly 400 pods, this settles into a rolling restart loop that never lets the service reach steady state. 09:40:00 — customer-visible latency and error rate both elevated due to constant pod churn; SLO alert fires (later than ideal, since aggregate capacity technically stayed adequate for a while). 10:05:00 — responder identifies the pattern via restart-count metrics and correlates to liveness probe failures specifically, not application errors. 10:15:00 — liveness probe timeout raised from 1s to 5s and failure threshold from 3 to 5 as an emergency change; churn stops within 10 minutes.

Root cause

The liveness probe's timeout was set well below the JVM's normal GC pause p99 (1.2s), which had been fine at lower traffic but became a real constraint once business-hours load pushed GC pause frequency and duration up. Liveness probes are meant to catch genuinely stuck processes, not brief GC pauses, and this one couldn't tell the difference.

The fix

We correlated the restart loop directly to probe failures with:

from logs
| where message == "k8s.pod.restarted" and service == "query-service"
| summarize restarts=count() by bin(time, 5m)
| where restarts > 20

Restart count peaked at over 60 pods every 5 minutes during the worst of it. We widened the liveness probe's tolerance and, more importantly, split liveness from readiness — the readiness probe now handles "temporarily busy, don't route new traffic here" while liveness only fires on genuine deadlock, checked with a much longer timeout.

What changed

Every JVM-based service now has its liveness probe timeout set with an explicit margin above its measured GC pause p99, reviewed whenever traffic patterns shift meaningfully. We also added pod-restart-rate as a first-class alert, since it turned out to be a leading indicator here that only fired after aggregate latency was already degraded.

  • Set liveness probe timeouts with real margin above measured GC pause p99, not a generic default.
  • Separate liveness (deadlock detection) from readiness (temporarily busy) — conflating them causes exactly this failure mode.
  • Alert on pod restart rate directly; it can lead aggregate latency/error signals during a churn loop.
  • Re-validate probe thresholds whenever a service's traffic pattern or JVM tuning changes.