This wasn't a single incident — it was three weeks of chasing a slow, creeping p95 regression on checkout until we found the connection-pool bug underneath it.

Checkout API p95 crept from 180ms to 310ms over three weeks in late January. No single deploy caused it, no alert fired, and it took a dedicated push to find and fix — bringing p95 back to 165ms, a 40%+ improvement over where it started.

Timeline

Week of Jan 5 — p95 sits at 180ms, stable. Week of Jan 12 — p95 drifts to 240ms; nothing pages because the SLO threshold is 400ms and error rate is unaffected. Jan 22 — a customer support ticket about "slow checkout" prompts Arjun to pull up the 30-day p95 trend and notice the drift. Jan 24 — team starts a focused investigation. Jan 26 — root cause identified: connection pool checkout wait time, not query time, is the growing component. Jan 28 — fix ships; p95 lands at 165ms.

Root cause

Our database connection pool was sized for a traffic pattern from eight months earlier. As we'd added two new checkout-adjacent services that also queried the same database, total concurrent connection demand had grown past the pool's max size, and requests were queueing for a connection before they ever issued a query.

from traces
| where service == "checkout-api" and span.name == "db.pool.acquire"
| summarize p50=p50(duration), p95=p95(duration) by bin(time, 1d)
| where p95 > 50ms

That query, run over the full 30-day window, showed pool-acquire p95 climbing from 8ms to 140ms — almost the entire regression was sitting in the "waiting for a connection" span, invisible if you only looked at query duration.

The fix

We separated the shared database into per-service connection pools with PgBouncer in transaction-pooling mode, raised the effective connection ceiling, and added explicit pool-utilization metrics per service. We also right-sized pool max based on measured p99 concurrent query count rather than a round number someone picked at launch.

What changed

The real lesson wasn't the pool sizing — it was that we had no alert on pool-acquire wait time, so a 17x regression grew for three weeks under an SLO that only watched total latency. We now track pool-acquire p95 as its own SLI with its own burn-rate alert, separate from end-to-end request latency.

  • Break end-to-end latency into its component spans and alert on each one, not just the total.
  • Size connection pools against measured concurrent demand, revisited quarterly, not a launch-day guess.
  • A support ticket is a valid incident trigger — don't wait for a threshold breach.
  • Per-service pooling isolates noisy neighbors from sharing one starved connection budget.