Our own rate-limiting service depended on a shared Redis instance that it also, ironically, helped saturate — a feedback loop that throttled legitimate traffic.

On July 6th, api-gateway began returning 429 Too Many Requests to customers who were nowhere near their actual limits. The rate limiter itself had gotten slow enough that its own retry behavior was consuming the Redis capacity it needed to answer quickly.

Timeline

15:30:00 — Redis, shared between the rate-limiter and two other services, experiences a brief latency increase from an unrelated large key scan run by one of those other services. 15:30:20 — rate-limiter calls to Redis start taking 40ms instead of 2ms; under load, this pushes some calls past their own 50ms timeout. 15:30:40 — on timeout, the rate-limiter's fail-open-with-retry logic retries the Redis call once before falling back to a conservative "deny" default if the retry also fails — a safety choice made to avoid accidentally allowing unlimited traffic during a Redis outage. 15:31:00 — with Redis already under elevated load, these retries add further load, causing more timeouts, causing more retries and more conservative denials — a self-reinforcing loop. 15:32:00 — 429 rate across normal, well-behaved traffic crosses 15%. 15:34:00 — on-call paged. 15:39:00 — responder identifies the Redis latency spike and the retry feedback loop via HelixQL, and temporarily fails the rate limiter fully open (allow all) while investigating. 15:41:00 — the original large key scan completes, Redis latency recovers, rate limiter returns to normal operation; fail-open override is removed at 15:50:00 once behavior is confirmed stable.

Root cause

The rate limiter's own retry-then-deny fallback was a reasonable safety default in isolation, but it turned a brief, unrelated Redis latency blip into a load-amplifying feedback loop, and the "deny" fallback meant the failure mode was maximally disruptive rather than gracefully degraded.

The fix

We traced the loop with:

from traces
| where service == "rate-limiter" and span.name == "redis.get"
| summarize p95=p95(duration), retries=countif(span.attempt > 1) by bin(time, 30s)
| where p95 > 40ms

We changed the fallback from "deny on Redis failure" to "fail open with a conservative in-memory local rate estimate" — allowing traffic through using a cached, slightly-stale limit rather than blocking outright, and removed the automatic retry entirely in favor of failing to the local estimate immediately on any timeout.

What changed

We also moved the rate-limiter's Redis usage to a dedicated instance, separate from other services, so an unrelated large scan can no longer create contention here. The core lesson we keep relearning: a safety default chosen to avoid one bad outcome (unlimited traffic) can create a worse one (a self-amplifying denial loop) under load — both need to be modeled, not just the one that sounds scarier.

  • Prefer fail-open-with-local-estimate over fail-closed/deny for rate limiters under backend failure — deny amplifies under load.
  • Remove automatic retries from any path that could turn backend slowness into a feedback loop.
  • Isolate shared infrastructure (like Redis) per critical consumer when contention risk is asymmetric.
  • Model both failure directions of a safety default, not just the one that sounds worse in the abstract.