A connection-draining setting that worked fine at low traffic caused real dropped requests once volume grew past what the old grace period could absorb.

On July 1st, a completely routine rolling deploy of api-gateway dropped roughly 1,400 requests over 90 seconds. The deploy process hadn't changed. Traffic volume had, quietly, over the preceding two months, past the point where the old drain timeout was safe.

Timeline

13:00:00 — a rolling deploy of api-gateway v8.9.0 begins, replacing pods in batches of 4 with a 15-second connection-drain grace period before SIGKILL, a setting unchanged for eight months. 13:00:15 — the first batch of 4 pods receives SIGTERM; in-flight requests get 15 seconds to complete before forced termination. 13:00:16 — at current traffic volume (up roughly 60% since this setting was last reviewed), some in-flight requests — particularly a slower reporting endpoint with a typical 18-second p95 — don't complete within the 15-second window. 13:00:30 — those in-flight requests are forcibly terminated mid-response, returning connection-reset errors to callers. 13:00:15–13:03:00 — this repeats across each deploy batch as the rollout proceeds, totaling roughly 1,400 failed requests. 13:03:00 — deploy completes; a post-deploy dashboard review (not a page — error rate stayed under alerting thresholds throughout) notices an anomalous connection-reset count.

Root cause

The 15-second drain grace period was set when the slowest endpoint on the gateway had a p95 around 8 seconds. As traffic and endpoint complexity grew, that ceiling crept up to 18 seconds for one specific endpoint, and nobody revisited the drain timeout because it wasn't tied to any monitored metric — it was just a value in a deploy config file.

The fix

We measured the actual gap with:

from traces
| where service == "api-gateway" and span.name == "http.request"
| summarize p95=p95(duration), p99=p99(duration) by route
| where p99 > 15000

One route stood out clearly at p99 18.4s, more than the drain grace period allowed. We raised the drain timeout to 25 seconds — comfortably above that route's measured p99 with margin — and added the drain timeout as a value that's checked against live endpoint latency automatically before each deploy, failing the deploy if any route's p99 exceeds 70% of the configured grace period.

What changed

Connection-drain settings across all services are now reviewed quarterly against current p99 latency, not left as a one-time deploy-time decision. We also added connection-reset count during deploys as a monitored metric on its own, since this incident stayed entirely under our existing error-rate alerting threshold despite being a real, customer-visible problem.

  • Tie connection-drain grace periods to measured endpoint p99 latency, not a static value set once at launch.
  • Automatically fail a deploy if any route's latency leaves insufficient drain margin.
  • Monitor connection-reset counts during deploys explicitly — they can hide under normal error-rate thresholds.
  • Revisit infra timeout constants quarterly as traffic and endpoint behavior evolve.