We had three people who could run a major incident and eleven who could not. Here is how we built a pipeline instead of a bottleneck.

Every serious incident routed to the same three people, because they were the only ones trusted to run it. That's not resilience, that's a single point of failure with a badge that says incident commander.

Shadow before you solo

We introduced a formal second-seat role: a trainee IC sits in on every declared incident above a certain severity, observes decisions in real time, and debriefs with the acting IC afterward without any responsibility for the outcome. This removed the all-or-nothing jump from zero incident experience to owning a live one.

Run drills on real, closed incidents

We replay a past incident's timeline with the trainee as IC and the original responders roleplaying their parts, using the actual burn-rate query and timeline data from the real event. It's realistic without the stakes, and gives us a query we can point trainees at directly.

from incidents
| where incident_id == "INC-4471"
| join (incident_timeline) on incident_id
| project _time, event, actor
| order by _time asc

Make it a track, not a favor

IC training used to happen informally, whoever a senior engineer decided to mentor. We made it a defined track with a checklist — number of shadowed incidents, number of drills, one supervised live incident — so opportunity doesn't depend on who happens to be paying attention to you. Headcount of qualified ICs went from three to nine in two quarters.

Rotate the on-call IC seat deliberately

Having nine qualified ICs doesn't help if the same three still get pinged out of habit whenever an incident channel opens. We added a dedicated IC-on-call rotation, separate from the engineering paging rotation, so incidents route to whoever is scheduled rather than whoever's name people remember first — the newest qualified ICs now run real incidents within weeks of finishing the track instead of waiting to be picked.

  • Introduce a formal second-seat / shadow role for every high-severity incident.
  • Run drills against real, closed incident timelines, not hypotheticals.
  • Define IC training as a checklist-based track, not informal mentorship.
  • Run a dedicated IC-on-call rotation so incidents don't default to old habits.
  • Track the number of qualified ICs as a resilience metric, not just page volume.