Fourteen Sev-1s in one quarter left the team running on fumes. Recovering wasn't about the next incident; it was about admitting we needed to stop first.

Fourteen Sev-1s in one quarter left the team running on fumes. Recovering wasn't about the next incident; it was about admitting we needed to stop first.

Noticing the Team, Not Just the Incidents

Each individual retro from that quarter looked fine on its own — root cause found, action items assigned, closed. What the retros didn't capture was the cumulative effect: the same six people had been in the room, or on the pager, for most of those fourteen incidents, and by week ten the quality of our incident response was visibly degrading, not because anyone got less skilled but because everyone was exhausted.

Measuring What We'd Been Ignoring

We pulled a per-engineer incident load for the quarter, something we'd never bothered to track because it felt like a distraction from "real" reliability metrics:

from incidents
| where created_at >= "2026-01-01" and created_at < "2026-04-01"
| summarize incidents = count(), total_hours = sum(duration_hours) by responder
| sort by total_hours desc

Two engineers had absorbed 60% of total incident hours for the quarter, both because they had the deepest system knowledge and because that expertise kept getting reinforced by them being the ones pulled into every ambiguous incident, which meant nobody else got the chance to build the same depth. It was a self-reinforcing bottleneck we'd built by rewarding competence with more load.

What Recovery Actually Looked Like

We didn't fix this with a motivational talk. We mandated a two-week rotation off any on-call or incident-response duty for the two most-loaded engineers, redistributed their usual incidents across the rest of the team even though it meant slower resolution times short-term, and started deliberately pairing less-experienced engineers onto ambiguous incidents specifically to spread the tribal knowledge instead of routing around it. Resolution time did get worse for about six weeks. Then it recovered, because five more people now had real experience with the hardest incident classes instead of two.

A team that always routes hard problems to its two most experienced people isn't building resilience — it's building a single point of failure with extra steps.
  • Track incident hours per engineer, not just per incident — concentration is invisible in per-incident retros.
  • Deep expertise concentrated in one or two people is a bottleneck, however good it feels in the moment.
  • Deliberately routing hard incidents to less-experienced engineers costs short-term speed and buys long-term resilience.
  • Mandate recovery time for over-loaded responders; "they'll say something if it's too much" doesn't work in practice.