A health-check misconfiguration marked two of three availability zones as unhealthy simultaneously, concentrating all traffic — and all risk — onto one AZ.

For 34 minutes on May 15th, 100% of production traffic was served from a single availability zone, after a health check misfire marked the other two as unhealthy. Nothing broke visibly, which is exactly why it took so long to notice.

Timeline

11:05:00 — a config change updates the load balancer's health-check path from /healthz to /healthz/deep, a more thorough check that also verifies downstream dependency connectivity. 11:05:30 — the deep check has a lower timeout (2s) than its actual p99 latency (2.4s) under normal load in two of three AZs, due to a slightly higher baseline latency to a shared dependency from those zones. 11:06:00 — those two AZs start failing health checks intermittently, then consistently as the load balancer's failure-counting logic accumulates consecutive failures. 11:08:00 — both AZs are marked unhealthy and drained; all traffic shifts to the one AZ whose path to the dependency happened to be faster. 11:08–11:39 — the single AZ handles 3x its normal share of traffic without visible degradation, because it had headroom, so no alert fired on error rate or latency. 11:39:00 — a capacity-utilization review (unrelated, routine) notices the traffic skew and escalates. 11:44:00 — health-check timeout raised to 4s; the two AZs recover and rejoin.

Root cause

The new health check's timeout was tuned against a single AZ's p99 during testing, not validated across all three, and the load balancer had no independent alert on AZ traffic distribution — only on aggregate error rate and latency, both of which stayed healthy throughout because the single serving AZ absorbed the load without complaint.

The fix

The skew was clear once we looked at distribution rather than aggregates:

from traces
| where span.name == "http.request"
| summarize count() by az, bin(time, 5m)
| where az != "us-east-1a"

Two AZs showed a hard drop to zero at 11:08 exactly. We raised the health-check timeout to account for the slower path's real p99 plus margin, and added an explicit alert on traffic-distribution skew across AZs, independent of whether aggregate metrics look fine.

What changed

We now require any health-check change to be validated against p99 latency in every AZ it applies to, not just one, before rollout. Single-AZ traffic concentration is now a paged alert on its own, because "everything looks fine in aggregate" is exactly the condition under which this failure mode hides.

  • Validate health-check timeouts against p99 latency in every AZ, not just the one you tested in.
  • Alert on cross-AZ traffic distribution directly — aggregate error rate and latency can both look perfectly healthy during a concentration event.
  • Remember that "nothing looks broken" during reduced redundancy is still an incident — it's latent risk, not safety.
  • Route routine capacity reviews to also flag anomalous traffic skew, not just absolute utilization.