A service mesh cert rotation left old and new root CAs out of sync across sidecars for four minutes, and every cross-service call in that window failed the handshake.

On June 24th, a scheduled quarterly rotation of our service mesh's root CA caused a four-minute window where sidecars presenting the new certificate couldn't be verified by sidecars still trusting only the old root — because the trust bundle rollout hadn't finished propagating before the leaf certificates started rotating.

Timeline

02:00:00 — the mesh control plane begins distributing the new root CA to all sidecar trust bundles, expected to take about 90 seconds fleet-wide via the existing config-propagation mechanism. 02:01:30 — the control plane, following its scheduled rotation plan, begins issuing new leaf certificates signed by the new root to a subset of services, assuming trust bundle propagation had completed. 02:01:45 — roughly 12% of sidecars had not yet received the updated trust bundle due to a propagation delay on nodes under higher load. 02:02:00 — mTLS handshakes between a new-leaf-cert sidecar and an old-trust-bundle sidecar begin failing, since the old bundle doesn't yet recognize the new root as trusted. 02:02:00–02:06:00 — cross-service call error rate rises to roughly 4% fleet-wide, concentrated on the affected node subset. 02:04:00 — SLO alert fires. 02:07:00 — trust bundle propagation catches up naturally as the delayed nodes finish updating; error rate returns to baseline without manual intervention, but not before a 6-minute window of degraded calls.

Root cause

The rotation plan didn't wait for confirmed trust-bundle propagation completion across 100% of sidecars before beginning leaf certificate issuance — it used a fixed 90-second delay as a proxy for "propagation is done," which held for 88% of the fleet but not the slower 12%.

The fix

We identified the affected node subset with:

from logs
| where message == "mtls.handshake_failed" and reason == "unknown_ca"
| summarize count() by dest_node, bin(time, 30s)
| sort by count desc

The failures clustered precisely on the nodes with delayed propagation. We changed the rotation process to require an explicit propagation-confirmed signal from every sidecar — not a fixed timer — before leaf certificate issuance begins for the new root.

What changed

Rotation now proceeds in a strict order: distribute new root to all sidecars, confirm 100% acknowledgment via an explicit health check, only then begin leaf issuance, and old roots stay trusted for a full rotation cycle afterward as overlap margin rather than being dropped immediately. We treat any future rotation plan relying on a fixed timer as a review flag.

  • Certificate rotation should gate on confirmed propagation, never a fixed timer.
  • Keep overlapping trust for old and new roots for a full cycle margin, not a tight handoff.
  • Alert on mTLS handshake failure reason codes specifically — "unknown_ca" points straight at a rotation issue.
  • Treat any "assume it's done after N seconds" step in an infra runbook as a latent risk worth removing.