Eleven minutes into my first incident as commander I realized nobody was going to tell me what to do next. That was the point.

Eleven minutes into my first incident as commander I realized nobody was going to tell me what to do next. That was the point.

The Job Isn't Fixing It

I'd been on-call for eight months before I got tapped to run point on a Sev-1: our ingest pipeline was backing up and write latency to the metrics store had crossed 30 seconds. My instinct was to dive into the query planner logs myself. A more senior engineer pulled me aside on the bridge and said the incident commander's job isn't to fix it — it's to make sure the right people are fixing the right thing, and everyone else knows what's happening. That reframing changed the whole hour.

Delegating Without Losing the Thread

I assigned one engineer to the ingest queue, one to check recent deploys, and I stayed on the timeline, posting an update every five minutes to the incident channel even when the update was "still investigating." The queue engineer found it fast:

from metrics
| where name == "ingest.queue.depth"
| where service == "ingest-writer"
| summarize max(value) by bin(5m)
| where value > 100000

Queue depth had crossed 100k events at 14:02, right after a config push raised our batch flush interval from 2s to 20s to "reduce write amplification." It did reduce write amplification. It also meant every write sat in memory twenty times longer, and under load that queue never drained. Reverting the flush interval brought depth back to baseline in six minutes.

What I'd Do Differently

I under-communicated in the first ten minutes because I was still deciding whether this was "real." That cost us: two other engineers had already independently found the config push and started separate threads before I'd even opened the incident channel. The fix was found three times over by people who couldn't see each other's work. Next time, I open the channel the moment I'm not sure, not the moment I'm certain.

  • Open the incident channel on suspicion, not confirmation — false starts are cheap, silent duplicate work is not.
  • The commander's output is a shared understanding of state, not a fix.
  • Assign explicit owners for investigation threads so people stop converging on the same log line.
  • A five-minute update cadence, even with nothing new to say, keeps the org from escalating out of anxiety.