Too little headroom and a normal traffic spike becomes an incident. Too much and you're burning budget on idle capacity. We found a number that works for us.

"Add more headroom" is not a capacity plan. We needed an actual number, derived from how spiky our real traffic is, not a round figure someone picked once.

Derive headroom from your actual spike ratio

We computed the ratio of our worst sustained 15-minute peak to our typical weekday average over the last two quarters, per service, instead of applying a flat 30% headroom rule everywhere. Some services spike 1.4x average; our billing-webhook consumer spikes 6x during month-end batch runs. One headroom number was never going to fit both.

from metrics("requests.count")
| where service == "billing-webhook-consumer"
| summarize avg_rps = avg(rate) by bin(_time, 15m)
| summarize typical = percentile(avg_rps, 50), peak = percentile(avg_rps, 99)
| extend spike_ratio = peak / typical

Headroom is a moving target, not a one-time budget line

We recompute spike ratio quarterly rather than setting capacity once at launch and forgetting it. A service that grows its user base gradually can see its spike ratio shift meaningfully within two quarters, and capacity that was generous at launch becomes tight without any single dramatic change to notice.

Alert on headroom shrinking, not just on saturation

By the time a saturation alert fires, you're already out of runway. We added a second, quieter alert that fires when the gap between current peak usage and provisioned capacity drops below 20%, giving the team weeks of lead time to provision instead of hours.

Price the headroom, don't just provision it

Extra capacity isn't free, and treating headroom as a purely technical decision hides its real cost from the people who'd want to weigh in on it. We now attach an estimated monthly cost to each service's headroom target next to its spike ratio, so a request to raise a target from 40% to 80% is visibly also a request for a specific dollar figure, not an abstract safety margin nobody has to account for.

  • Derive headroom per service from its actual peak-to-typical spike ratio.
  • Recompute the ratio quarterly — it drifts with organic growth.
  • Alert on shrinking headroom, not only on saturation itself.
  • Attach an estimated cost to headroom targets so the tradeoff is visible.
  • Treat a flat, org-wide headroom percentage as a starting guess, not a final answer.