A partner's webhook sender retried aggressively on our 5xx responses, and their retries became the majority of load on the endpoint they were retrying against.

On May 1st, our webhooks-inbound endpoint had a brief database blip that caused about 90 seconds of 5xx responses to one partner's webhook deliveries. Their retry logic then sent us 14x normal volume for the next 25 minutes, which kept the endpoint degraded long after our original problem had cleared.

Timeline

16:20:00 — a brief database failover causes webhooks-inbound to return 500 for about 90 seconds to all callers, including one high-volume partner. 16:21:30 — our database recovers and would normally serve traffic fine again. 16:22:00 — the partner's system, seeing 500s, begins retrying every failed webhook with what turns out to be immediate retries at 3 attempts, no backoff, applied to their entire batch of ~2,000 webhooks queued in that window. 16:23:00 — inbound request volume from that partner is 14x baseline; the endpoint, sized for their normal volume plus headroom, starts queueing again — this time from the retry storm itself, not the original database issue. 16:24:00 — on-call paged again on renewed 5xx rate. 16:30:00 — responder identifies the traffic source via HelixQL, rate-limits that partner's source IP specifically rather than the endpoint globally, and lets the queue drain. 16:47:00 — retry volume tapers as the partner's own retry budget exhausts; endpoint fully recovers.

Root cause

Our original 90-second blip was real but brief and self-resolved. The extended 27-minute impact was entirely a self-inflicted amplification from a partner's retry policy that had no backoff and applied to an entire backlog at once rather than pacing retries.

The fix

We isolated the partner's traffic share with:

from logs
| where service == "webhooks-inbound" and time > ago(1h)
| summarize count() by partner_id, bin(time, 1m)
| sort by count desc

One partner_id accounted for over 90% of requests in the affected window, versus its usual 8% share. We applied a per-partner rate limit as an immediate fix and reached out to the partner about their retry policy afterward.

What changed

We now enforce per-partner rate limits on all inbound webhook endpoints by default, sized to roughly 3x each partner's baseline — enough headroom for legitimate bursts, not enough to let one partner's retry storm consume the whole endpoint's capacity. We also publish explicit retry guidance (max 3 attempts, exponential backoff starting at 1s) in our partner integration docs.

  • Rate-limit inbound integrations per partner by default, not just globally.
  • Publish explicit retry guidance to partners and reference it in onboarding, not just in fine print.
  • Distinguish "our fault, self-resolved" from "amplified by someone else's retry policy" in the retro — the fixes are different.
  • A brief internal blip can have a long external tail; measure total incident duration, not just root-cause duration.