A single Redis restart evicted a hot key that three unrelated services all depended on. All three fell back to the database within the same second.

On February 3rd, a routine Redis primary failover cleared a shared cache. The key that mattered most — a feature-flag config used by product, checkout, and pricing services — expired at the same instant for all three, and all three stampeded the config database simultaneously.

Timeline

08:58:00 — scheduled Redis maintenance triggers a primary failover; the replica takes over but starts cold. 08:58:05 — the "active-feature-flags" key, normally cached with a 5-minute TTL and read by roughly 40 requests per second across three services, is now a cache miss for every reader. 08:58:06 — all three services independently fall back to querying the config Postgres database directly, with no request coalescing. 08:58:10 — config database connections spike from a baseline of 15 to over 400; query latency there goes from 4ms to 3.2s. 08:59:00 — product, checkout, and pricing services all start returning 503s as their own request queues back up waiting on the config lookup. 09:02:00 — on-call restores service by manually warming the cache key and enabling emergency rate limiting on config-db reads. 09:06:00 — full recovery.

Root cause

No service used request coalescing (a "single-flight" pattern) for cache misses on this key, so a cold cache turned N cache-missing requests per second into N simultaneous database queries instead of one query plus N requests waiting on it. Three services sharing the same hot key multiplied the effect three times over.

The fix

We found the fan-out pattern with a HelixQL query joining traces across services on the same downstream call:

from traces
| where span.name == "config.get_active_flags" and span.attributes["cache.hit"] == "false"
| summarize misses=count() by service, bin(time, 5s)
| where misses > 20

All three services lit up in the same 5-second bucket. We implemented single-flight coalescing in the shared client library so only one in-flight database query exists per key across all callers, with everyone else awaiting its result.

What changed

Cache warm-up now runs automatically after any Redis failover for keys flagged "hot" (read >10 req/s), before traffic is allowed to hit them cold. We also staggered TTLs with jitter so shared keys don't expire at the exact same moment across services even without a failover trigger.

  • Add single-flight request coalescing to any client library backing a shared hot cache key.
  • Jitter TTLs on shared cache keys so they don't expire in lockstep.
  • Auto-warm known-hot keys immediately after any cache failover, before serving live traffic.
  • Alert on cache miss rate per key, not just overall hit ratio — an aggregate hides single-key stampedes.