An engineer's config typo caused a real outage. Here is how we ran the retro without naming a villain, and what the process looked like from the inside.

On June 2nd, a single-character typo in a rate-limit config took the search API down for eight minutes. We're writing this one up not for the technical fix, which is small, but for how the retro itself was run — because the process is the part worth reusing.

Timeline

10:12:00 — an engineer updates a rate-limit config, intending to set the search-api limit to 1000 requests/second; a typo sets it to 100 instead. 10:13:00 — the config deploys through the normal pipeline, no review flag, since a lower rate limit isn't the kind of change existing guardrails watch for. 10:13:30 — search-api begins rejecting roughly 90% of requests with 429s, since real traffic is running at approximately 950 req/s. 10:14:10 — SLO burn-rate alert fires. 10:16:00 — on-call identifies the config change as the cause via the deploy timeline within two minutes. 10:20:00 — corrected config deploys. 10:21:00 — full recovery.

Root cause

A single-digit typo in a numeric config value, with no automated bounds-checking against recent traffic levels and no second reviewer required for config-only changes below a certain "risk" classification that this change didn't meet.

The fix

The retro itself started with the data, not the narrative — we pulled the exact request-rejection curve before anyone described what happened:

from traces
| where service == "search-api" and status_code == 429
| summarize count() by bin(time, 15s)
| where count() > 100

We added bounds-checking to rate-limit config changes: any value more than 20% below the trailing 7-day p95 traffic level triggers a required second approval before deploy. This catches typos and unit-confusion errors without slowing down legitimate limit reductions.

What changed

The retro document itself never names the engineer, refers only to "the change" and "the pipeline," and every corrective action targets the system (bounds-checking, review gates, better default protections) rather than "be more careful." The engineer who made the typo co-wrote the retro and proposed the bounds-check fix themselves — which is, in our experience, what a functioning blameless process actually produces: the person closest to the gap is usually the best person to close it, once they're safe to speak up.

  • Bounds-check config changes against recent real traffic automatically; don't rely on manual review to catch numeric typos.
  • Write the retro from data first, narrative second — pull the graphs before writing the story.
  • Never name individuals in a retro document; describe systems, changes, and gaps.
  • Invite the person closest to the incident to help write the fix — they usually have the sharpest read on it.