A payment-gateway blip should have been a five-second blip. Instead, client-side retries turned it into an 11-minute checkout outage. Here is the timeline.

At 14:02 UTC on January 12th, our payment gateway partner had a 4-second latency spike on their side. By 14:04 our own checkout API was returning 5xx to 60% of requests. The gateway blip lasted seconds. Our outage lasted eleven minutes.

Timeline

14:02:10 — upstream payment gateway p99 jumps from 220ms to 4.1s for about 40 seconds. 14:02:41 — our checkout-service client, configured with a 2s timeout and 3 retries with no backoff, starts re-sending the same requests immediately on timeout. 14:03:15 — outbound request volume to the gateway is 6x normal; the gateway starts shedding load and returning 429s, which our client also retries immediately. 14:04:02 — checkout-service's own connection pool (200 connections) is exhausted waiting on retries; new incoming checkout requests start queueing. 14:04:30 — on-call is paged on checkout 5xx rate > 5%. 14:09:00 — responder identifies the retry amplification via HelixQL and disables the retry middleware via feature flag. 14:13:00 — 5xx rate back under 0.5%.

Root cause

The retry policy had no backoff and no jitter, and worse, it did not distinguish between a timeout (retry-worthy) and a 429 (do-not-retry-immediately). A single 40-second upstream blip turned into a 6x traffic multiplier that outlasted the original problem by 9 minutes because the retries themselves were now the load.

The fix

We found the amplification pattern by comparing inbound request volume to outbound gateway calls per trace_id — a 1:3 ratio should be rare and a 1:6 ratio should almost never happen outside an incident.

from traces
| where service == "checkout-api" and span.name == "pay.charge"
| summarize calls=count(), retries=countif(span.attempt > 1) by bin(time, 30s)
| extend retry_ratio = retries * 1.0 / calls
| where retry_ratio > 0.4

That query, re-run against the 14:00–14:15 window, showed the retry ratio crossing 0.55 at 14:03:00 — a full minute before the connection pool exhaustion paged anyone. We capped retries at 2, added exponential backoff with jitter (250ms base), and made 429 responses non-retryable for 5 seconds via a client-side circuit breaker.

What changed

We now alert directly on retry_ratio > 0.3 sustained for 20 seconds on any service, which would have paged us a full minute earlier than the 5xx-rate alert did. We also added a standing HelixQL saved query, "retry-amplification," to the on-call dashboard so responders don't have to write it from scratch under pressure.

  • Cap client retries at 2 with exponential backoff and jitter on all payment-path services.
  • Treat HTTP 429 as non-retryable for a cooldown window, not an automatic retry trigger.
  • Alert on retry_ratio > 0.3, not just downstream error rate — it fires earlier.
  • Load-test the retry policy itself during game days, not just the happy path.