A scaling policy tuned for a smooth traffic curve met a genuinely spiky workload and oscillated between 8 and 40 pods for six straight hours.

From 09:00 to 15:00 on April 23rd, ingest-processor's pod count oscillated wildly — scaling up to 40, back down to 8, repeatedly — instead of settling at a stable size. Latency was fine on average, but p99 spiked every time a scale-down killed pods mid-burst.

Timeline

09:00:00 — a new customer's traffic pattern goes live: short, sharp bursts of ingest volume every 4–5 minutes rather than the smooth curve most customers produce. 09:02:00 — CPU-based autoscaling reacts to each burst by scaling up, then scales back down within the policy's 2-minute cooldown once the burst passes, only for the next burst to trigger scale-up again 2 minutes later. 09:00–15:00 — this repeats roughly 70 times, with each scale-down killing pods that were mid-processing, causing retries and p99 spikes to 1.8s during each transition. 14:50:00 — an SRE reviewing the weekly capacity dashboard notices the oscillation pattern (not paged — nothing breached a hard threshold). 15:10:00 — team manually raises the minimum pod floor to 25 and extends cooldown to 10 minutes as an immediate mitigation. 15:15:00 — oscillation stops.

Root cause

The autoscaling policy's cooldown window was shorter than the customer's burst interval, so the system never had a chance to observe a stable state before reacting to the next burst. CPU as the sole scaling signal also meant it was reacting to the burst's peak, not its sustained load.

The fix

We visualized the oscillation with:

from metrics
| where metric.name == "k8s.pod.count" and service == "ingest-processor"
| summarize count() by bin(time, 1m)
| serialize
| extend delta = count() - prev(count())
| where abs(delta) > 5

70 scale events in 6 hours, each swinging by 15+ pods, confirmed the flap pattern precisely. We switched the scaling signal from instantaneous CPU to a 5-minute moving average, and extended the cooldown to 10 minutes — long enough to ride out a single burst without reacting.

What changed

We also added a "min stable floor" concept per service, set from the p95 sustained load over the trailing 7 days rather than a static number, so bursty customers don't force the whole service into a fragile minimum. Pod-count oscillation itself is now a monitored signal with its own alert, so we catch the next one without waiting for a manual dashboard review.

  • Use moving-average signals, not instantaneous ones, for autoscaling triggers on bursty workloads.
  • Set cooldown windows longer than the expected burst interval, not a generic default.
  • Alert on pod-count oscillation itself as a first-class signal, not just on latency or CPU thresholds.
  • Set scaling floors from measured sustained load, revisited as traffic patterns change.