A regional spot-price spike reclaimed a third of our worker fleet within 90 seconds, and our scale-up response was too slow to absorb the sudden capacity cliff.

On July 3rd, a spot-price spike in one region triggered simultaneous reclamation of 140 of our roughly 420 spot-backed worker nodes within a 90-second window. Job processing latency spiked hard while replacement capacity caught up.

Timeline

21:14:00 — a regional demand spike from other cloud tenants pushes spot prices above our bid ceiling on a specific instance family. 21:14:10 — the cloud provider begins sending 2-minute reclamation notices to affected instances; because we'd concentrated a large share of workers on this one instance family for cost reasons, 140 nodes receive notices nearly simultaneously. 21:16:10 — reclamation completes; 140 nodes terminate within the same short window, a third of total worker capacity gone at once. 21:16:15 — job queue depth begins climbing sharply as remaining capacity can't keep pace. 21:17:00 — autoscaling begins provisioning replacements, but on-demand and alternate spot instance-family launches take roughly 3–4 minutes to become ready. 21:18:30 — SLO alert fires on job processing latency. 21:21:00 — first replacement capacity comes online. 21:24:00 — queue depth peaks, then begins draining. 21:34:00 — full recovery.

Root cause

We had diversified across availability zones but not across instance families, so a single spot-price event affecting one family could reclaim a disproportionate share of total capacity simultaneously — the reclamation notices themselves were standard and expected, but our concentration in one family turned an individually-normal event into a fleet-wide capacity cliff.

The fix

We confirmed the concentration risk with:

from metrics
| where metric.name == "k8s.node.count" and node.lifecycle == "spot"
| summarize count() by instance_family, bin(time, 5m)

One instance family accounted for 68% of spot capacity going into the incident. We diversified across four instance families with automatic rebalancing, capped any single family at 30% of total spot capacity, and kept a larger standing on-demand floor (from 10% to 20% of total capacity) specifically to absorb reclamation cliffs without a latency spike while replacements spin up.

What changed

We also shortened our scale-up path for replacement capacity by pre-warming a small buffer of ready-to-join on-demand instances rather than launching from zero on every reclamation event, cutting replacement time from 3–4 minutes to under 90 seconds. Instance-family concentration is now a monitored metric with an alert if any single family exceeds its 30% cap.

  • Diversify spot capacity across instance families, not just availability zones — cap any single family's share.
  • Keep a larger on-demand floor sized to absorb a full reclamation cliff, not just routine scaling noise.
  • Pre-warm standby replacement capacity to cut scale-up latency after a reclamation event.
  • Monitor capacity concentration by instance family as its own alertable metric.