A routine node pool upgrade evicted pods faster than our disruption budget accounted for, and three runbook commands stopped working mid-incident.

A routine node pool upgrade evicted pods faster than our disruption budget accounted for, and three runbook commands stopped working mid-incident.

Two Things Happening at Once

We had a real incident already in progress — elevated latency on the trace-ingestion path — when the platform team's scheduled node pool upgrade kicked in, evicting pods across the cluster to migrate them to new nodes. Individually, either event would have been routine. Together, the eviction wave hit our already-stressed ingestion pods right as we were trying to scale them up to handle the original problem, and our pod disruption budget wasn't tight enough to prevent both happening simultaneously.

Diagnosing Two Root Causes at Once

The confusing part was that our scale-up commands kept appearing to succeed and then immediately losing pods again. We had to separate the two effects:

from logs
| where namespace == "ingestion"
| where message contains "pod evicted" or message contains "scaled up"
| summarize count() by bin(2m), message
| sort by bin(2m) asc

The timeline showed scale-up events and eviction events interleaved almost one-for-one for about eighteen minutes — we were adding pods and the node upgrade was removing them at nearly the same rate, so our capacity never actually grew even though every individual command reported success. It took explicitly correlating against the platform team's upgrade log, which we hadn't thought to check because the original incident felt self-contained, to realize we were fighting two problems that looked like one.

What We Changed

We set a tighter pod disruption budget for ingestion-critical services so voluntary evictions can't exceed a small percentage at once, and we added a check to our incident-start checklist: query for any in-progress infrastructure changes (node upgrades, scheduled maintenance) across the whole cluster, not just our own service, in the first five minutes of any incident. It's a small addition to the runbook that would have saved us the eighteen confusing minutes entirely.

  • Scheduled infrastructure changes can silently compound an unrelated incident — check for them explicitly and early.
  • A tight pod disruption budget matters most exactly when you least expect to need it: mid-incident.
  • "Command succeeded" doesn't mean "state improved" when something else is undoing it in parallel.
  • Cross-team change visibility (even read-only) should be a first-five-minutes checklist item, not an afterthought.