A well-meaning incident commander made a rough call that extended an outage by twenty minutes. The retro that followed rebuilt how we train the role entirely.

He wasn't careless. He was newly certified, following the checklist exactly as trained, and the checklist was the problem — it told him what to do at each step but never taught him when to break from it, and the incident needed exactly that.

What the checklist couldn't teach

Our incident commander training at the time was almost entirely procedural: assign roles, open a channel, set a cadence, escalate at defined thresholds. What it didn't cover was judgment under ambiguity — specifically, when the standard cadence of "update every 15 minutes" actively works against you, because in this incident, waiting for the 15-minute mark to check a hypothesis meant a fix sat ready and untested for ten extra minutes.

What we added to training

We built a second training phase focused entirely on deviation — recognizing when the procedure is actively wrong for the situation in front of you, not just executing it faithfully.

ic_training_phase_2: judgment_under_procedure
format: paired simulation with a senior IC as silent observer
scenario_design: >
  Every scenario has at least one point where following the standard
  checklist step-by-step is measurably worse than deviating from it.
  Trainee is not told this in advance.
debrief_questions:
  - Where did you follow the checklist because it was right, vs.
    because it was the checklist?
  - What signal would have told you to deviate, and did you see it?
graduation_criterion: >
  Trainee must correctly identify and act on at least 2 of 3
  deviation points across their simulation set.
"I'd been trained to run the process. Nobody had trained me to notice when the process was the thing slowing us down."

What changed for that engineer, specifically

He went through the new training himself, and six months later ran an incident where he explicitly skipped a scheduled status update to ship a fix immediately — the exact judgment call the old training would never have prepared him to make with confidence. We didn't discipline him for the original incident. We fixed the gap that made his response predictable, not personal.

  • Procedural incident-commander training teaches execution but not judgment — the two need to be trained separately.
  • Design training scenarios where following the checklist exactly is measurably wrong, so trainees practice recognizing deviation points.
  • A rough incident-commander call is usually a training gap, not a personal failing — fix the gap, not just the person.