A load balancer routing rule silently excluded the canary pool for two releases in a row, so two broken deploys sailed through with zero canary signal.

On April 2nd we discovered that our canary deployment process had been validating nothing for the previous two releases. A routing rule change three weeks earlier had accidentally excluded canary-tagged pods from receiving any production traffic at all.

Timeline

This one wasn't caught by an alert — it surfaced when a responder investigating an unrelated latency question noticed the canary pool's request count metric had read zero for three weeks. 09:10:00 — Priya, debugging a different issue, queries canary pod request volume as a sanity check and finds it flatlined at zero since March 12th. 09:20:00 — cross-referenced against the load balancer config history, a routing weight change on March 12th (meant to adjust regional traffic split) had an unrelated side effect: it reset the canary target group's weight to 0 and nobody caught it because the deploy pipeline doesn't verify canary traffic receipt before promoting. 09:35:00 — team confirms the last two "successful" canary-gated releases (March 19th and March 26th) had in fact received zero live validation before full rollout.

Root cause

The canary promotion gate checked that the canary pods were healthy and running, not that they had actually received production traffic. A routing misconfiguration could satisfy "pods are up" while completely failing "pods are serving requests," and the gate had no way to distinguish the two.

The fix

We added a hard precondition to the canary promotion pipeline using this check:

from traces
| where pod.role == "canary" and service == "checkout-api"
| summarize requests=count() by bin(time, 5m)
| where requests < 50

If canary request volume over the evaluation window falls under a minimum floor, promotion is now blocked automatically with an explicit "no traffic received" failure, rather than silently passing because nothing errored.

What changed

We retroactively reviewed the two releases that skipped real canary validation — both turned out to be benign, but we treated that as luck, not vindication. Load balancer routing-weight changes now require a post-change verification step confirming traffic distribution matches intent, checked automatically rather than trusted by inspection of the config diff alone.

  • Canary gates must verify traffic receipt, not just pod health — the two are not the same thing.
  • Any load-balancer routing change needs an automated post-change traffic-distribution check.
  • Treat "we got lucky" and "the safety net worked" as different outcomes, even when the release was fine.
  • Periodically audit canary/staging traffic volume as a standing health check, not just during incidents.