Not every retro is about what went wrong. This one is about what went right: a bad deploy caught and rolled back in under nine minutes end to end.

On January 16th a routine deploy to notification-service shipped a broken readiness probe. Total customer impact: nine minutes, and only on retried background jobs. We think it's worth writing up precisely because it went well.

Timeline

10:41:00 — v2.44.0 of notification-service rolls out to 20% of pods via canary. 10:41:30 — the new build's readiness endpoint returns 500 because of an unhandled nil config value, but the pods stay in the load balancer for 30 seconds past the failed check due to a slow deregistration setting. 10:42:00 — error rate on notification-service crosses 8%, our SLO burn-rate alert fires (2% of the 30-day error budget in under 5 minutes). 10:42:40 — on-call Lena is paged, opens the incident channel, and pulls up the deploy timeline panel. 10:44:10 — she confirms the correlation between the v2.44.0 rollout marker and the error spike using HelixQL, and triggers rollback. 10:45:50 — canary pods are drained and replaced with v2.43.1. 10:49:30 — error rate back to baseline.

Root cause

A config struct that used to be populated by a required environment variable silently defaulted to nil when we switched to a new config-loading library two sprints earlier. It only surfaced when the readiness probe path touched that specific field, which the pre-deploy smoke tests didn't exercise.

The fix

The burn-rate alert is what made this fast — it fired on budget consumption velocity, not a static error-rate threshold, so it caught the spike in under 90 seconds. The confirming query took 90 more seconds:

from deploys
| where service == "notification-service"
| join (from traces | where service == "notification-service" and status == "error") on 1==1
| where trace.time between (deploy.time, deploy.time + 10m)
| summarize errors=count() by deploy.version, bin(trace.time, 30s)

The output made the v2.44.0-only error spike undeniable, which meant no debate, no "let's watch it for a few more minutes" — just rollback.

What changed

We added a required-field validation step to the config loader's startup path so a nil-default fails fast at boot instead of at first use, and we shortened canary deregistration from 30s to 5s.

  • Burn-rate alerts beat static thresholds for catching bad deploys fast — keep tuning them, not replacing them.
  • Keep a one-query "what changed in the last 10 minutes" HelixQL saved search on every service dashboard.
  • Fail-fast config validation at boot, not at first access.
  • Shorten canary deregistration windows; 30 seconds is too long to matter at this traffic volume.