Every request started failing at once, not gradually. That's the signature of a hard resource limit, not a slow degradation, and it changes how you look.

Every request started failing at once, not gradually. That's the signature of a hard resource limit, not a slow degradation, and it changes how you look.

A Cliff, Not a Slope

Most of the incidents I write up have a ramp: latency climbs, error rate creeps, something degrades over minutes. This one didn't. Error rate on our orders-api went from 0.2% to 97% in under thirty seconds, and stayed there flat, which is a different diagnostic problem entirely — a cliff points at a hard limit being hit, not a resource slowly running out.

Finding the Limit

The error messages themselves said "could not obtain connection from pool," which at least pointed in a direction immediately. The question was why now, on a Tuesday night with unremarkable traffic:

from metrics
| where name == "db.pool.active_connections"
| where service == "orders-api"
| summarize max(value) by bin(1m)
| sort by bin(1m) desc
| limit 20

Active connections had been sitting comfortably at 40 out of a 50-connection pool limit for weeks, plenty of headroom. That night it hit 50 and stuck there. Cross-referencing with a deploy fifteen minutes earlier for an unrelated service — a reporting job that shared the same database — showed the reporting job's new version had removed a connection timeout that used to force idle connections closed after 30 seconds. Without it, the reporting job's connections accumulated instead of cycling, slowly eating into shared pool capacity from a completely different service until orders-api, the most connection-hungry consumer, hit the wall first.

Hard Limits Need Hard Alerts

The unsettling part is how little warning there was: connections at 40/50 for weeks looked like healthy headroom right up until the moment it wasn't, with no gradual slope to alert on. We added a threshold alert at 80% of pool capacity specifically because a percentage-of-limit metric like this behaves nothing like a typical gradual-degradation signal — it's fine until it's a cliff, so the alert has to fire well before the edge, not react to crossing it.

  • Sudden, flat failure (a cliff, not a ramp) points at a hard resource limit, not a slow degradation.
  • Shared resources (a connection pool, a rate limit) mean an unrelated service's change can exhaust your headroom.
  • "Plenty of headroom for weeks" can flip to zero headroom in seconds — alert on percentage-of-limit early, not late.
  • Check deploys across every service sharing the resource, not just your own, when a hard-limit failure shows up.