A brief database hiccup turned into a 40-minute outage because every client retried at once. Here is the anatomy of a retry storm and the four lines that stopped it.

The database was slow for eight seconds. The outage lasted forty minutes. The gap between those two numbers is a story about retries.

How a hiccup becomes a storm

When the database paused, every in-flight request failed and every client retried. Those retries arrived in a synchronized wave, hitting a database that was already recovering. It fell over again, producing another wave. We had built a feedback loop and pointed it at ourselves.

The three fixes

  • Exponential backoff — each retry waits longer than the last, so the waves spread out instead of stacking up.
  • Jitter — randomizing each client's wait breaks the synchronization that turns retries into a thundering herd.
  • A retry budget — cap retries as a fraction of total requests, so a struggling dependency is never hit with more load than it was already failing under.

What the fix looks like

delay = min(cap, base * 2 ** attempt)
sleep(random_between(0, delay))   # full jitter
if retries_this_minute > 0.1 * requests_this_minute:
    fail_fast()                   # protect the dependency

The dashboard we should have had

We now graph retry rate as a percentage of request rate on the same panel. A healthy service sits near zero. A retry storm shows up as that line climbing toward one — long before the outage that used to be our first clue.