A routine retro for a two-hour outage turned into the reason we rewrote our entire senior-engineer interview loop.

The outage itself was unremarkable — a bad config rollout, caught and reverted in 90 minutes. What wasn't unremarkable was the retro, where we discovered the person who diagnosed it fastest had been in the company for six weeks, and the person who most confidently pointed at the wrong cause had eight years of tenure and a senior title.

What the retro revealed about seniority

We went back through a year of incident timelines and tagged every hypothesis anyone had proposed during an active incident, correct or not, against their tenure and level. Tenure had almost no correlation with hypothesis accuracy. What correlated strongly was something we hadn't been screening for at all: willingness to say "I'm not sure, let's check" instead of committing early to a confident-sounding theory.

"The most dangerous person in an incident channel isn't the one who doesn't know. It's the one who sounds certain and is wrong." — retro notes, HX-3812

What we changed in the loop

We added a new interview stage specifically designed to surface overconfidence, not competence — the two had been getting conflated for years.

interview_stage: calibration_check
format: >
  Candidate is shown 5 short incident summaries with a plausible-but-wrong
  root cause already proposed by a fictional teammate. Ask candidate to
  rate their confidence (1-5) that the proposed cause is correct, then
  explain their reasoning either way.
scored_on:
  - does confidence rating correlate with actual correctness across the 5
  - does candidate ask clarifying questions before rating
  - does candidate revise their rating when given new information
red_flag: high confidence + no request for more data, more than twice

What it cost us and what it bought us

This stage adds fifteen minutes to an already long loop, and two candidates who otherwise would have received offers were screened out for consistent overconfidence with no revision under new information. In exchange, incident retros over the following two quarters showed fewer "we chased the wrong theory for 40 minutes before someone senior admitted doubt" entries — down from a recurring pattern in roughly a third of major incidents to under one in ten.

  • Confident and correct are not the same trait, and most interview loops only screen for confidence.
  • A calibration check — rating confidence against known-wrong plausible answers — surfaces overconfidence that technical questions miss entirely.
  • Watch your incident timelines for wasted time chasing a confidently-wrong early theory; it's a hiring signal, not just an incident detail.