Request volume tripled in ninety seconds with no traffic change upstream. That shape only means one thing: the system is retrying itself into the ground.

Request volume tripled in ninety seconds with no traffic change upstream. That shape only means one thing: the system is retrying itself into the ground.

The Graph That Wakes You Up Fast

A sawtooth in request volume with no matching change in unique users is the retry storm signature, and once you've seen it a few times your body reacts before your brain finishes parsing the graph. This time it was our recommendations service timing out against a downstream feature store, and every client had a retry-with-backoff policy that, individually, was reasonable and, collectively, was catastrophic.

Finding the Amplification Factor

The feature store wasn't down, just slow — p50 had gone from 12ms to 400ms after a schema migration added an unindexed lookup. That alone shouldn't take anything offline. But three layers of the call graph each retried up to three times on timeout, and none of them knew about the others:

from traces
| where service == "feature-store"
| where span.attributes["retry_count"] > 0
| summarize requests = count() by bin(1m), retry_count
| sort by bin(1m) asc

The query showed retry_count climbing from a baseline near zero to a distribution where a third of requests were on their second or third attempt. A single slow query at the bottom was turning into up to 27x amplification at the top of the stack (3 retries × 3 retries × 3 retries) before anything actually failed outright. The feature store, already slow, was now also drowning in duplicate work, which made it slower, which triggered more retries. Classic death spiral.

The Fix That Held

We killed the unindexed lookup with a targeted index — the actual root cause — but that took forty minutes to deploy safely. What stopped the bleeding immediately was disabling retries at the two outer layers via a feature flag, accepting a higher error rate for ten minutes in exchange for letting the feature store recover under its real load instead of 27x it. Error rate spiked to 8% briefly, which felt bad in the moment but was a fraction of the alternative.

Retries are a local optimization that becomes a global liability the moment more than one layer does it independently.
  • A volume sawtooth with flat unique-user counts is almost always retries, not real traffic.
  • Multiply retry budgets across your call graph before you ship them — three layers of 3x retries is 27x, not 9x.
  • A "disable retries" kill switch is cheap insurance and buys time for the real fix.
  • Index changes and schema migrations deserve a latency regression check before rollout, not after.