We tested a dozen interview questions against eighteen months of real incident-commander performance. Only one actually predicted anything.

We wanted to know, empirically, whether any of our standard interview questions predicted who would turn out to be a good incident commander once hired. So we went back and scored eighteen months of real incidents against the interview answers those same engineers had given.

What didn't predict anything

Technical depth questions — "walk me through how you'd debug a memory leak" — showed essentially no correlation with incident-commander performance. Neither did years of experience, nor confidence as rated by the interviewer. The strongest technical candidates were just as likely to freeze or fixate during a real incident as anyone else.

The one question that did

"Tell me about a time you had to make a decision with incomplete information, where you were wrong, and describe exactly how you found out." What predicted good incident commanding wasn't the decision itself — it was whether the candidate could narrate updating their belief cleanly, without defensiveness, and whether they described a concrete mechanism for finding out they were wrong rather than a vague "we figured it out eventually."

scoring_rubric: incident_commander_signal_question
prompt: >
  Tell me about a decision you made with incomplete information that
  turned out wrong, and how you found out.
score_high_if:
  - describes a SPECIFIC mechanism for discovering the error
    (a metric, a user report, a teammate — not "it became clear")
  - narrates the belief update without minimizing or over-apologizing
  - can state what they'd do differently in one sentence, not five
score_low_if:
  - claims to have never really been wrong, or reframes it as someone
    else's fault
  - the "finding out" mechanism is vague or absent
"The best incident commanders aren't the ones who are rarely wrong. They're the ones who notice fastest."

What we did with this

We moved this question to the front of the loop for anyone being considered for the incident-commander pool, and made it the single highest-weighted signal for that specific role — outweighing even the technical rounds. It's not a replacement for technical competence; it's a filter for a trait technical rounds can't see at all.

  • Technical depth and incident-commander skill are only weakly related — test for them separately.
  • A single well-designed question about noticing and correcting an error outperformed a dozen conventional ones in our data.
  • Validate your interview questions against real on-the-job outcomes periodically — most companies never check whether their questions predict anything.