Nobody joined the call. The whole incident got resolved in a thread, across three time zones, and it was our fastest response of the quarter.
Nobody joined the call. The whole incident got resolved in a thread, across three time zones, and it was our fastest response of the quarter.
Why We Even Tried This
Our team spans four time zones, and a mandatory video bridge for every Sev-2 was quietly punishing whoever happened to be awake at the worst hour. For a moderate-severity billing delay — invoices generating twenty minutes late, no customer-facing outage — we tried something different: a structured incident thread with a strict format instead of a call. Every message had to be a status update, a finding, or an action, tagged as such, no exceptions.
What the Thread Actually Looked Like
The engineer in Lisbon posted the trigger and pulled the first data point:
from logs
| where service == "billing-worker"
| where message contains "queue_lag"
| summarize max(lag_seconds) by bin(5m)
| sort by bin(5m) desc
| limit 10
Lag was climbing linearly, not spiking — a strong hint it was a throughput problem, not an outage. Six hours later, someone in Austin picked up the thread cold, read the last five messages, and had full context in under two minutes because every post was self-contained instead of assuming shared real-time knowledge. She found the actual cause: a batch job that reprocesses failed invoices had been silently doubling in size each week as failed invoices accumulated, and it was now competing for worker capacity with real-time processing.
Where Async Breaks Down
This only worked because the severity was low enough to tolerate a slower cadence and because we'd agreed in advance on the thread discipline — timestamped updates, one finding per message, explicit "I am now doing X" statements. For anything customer-facing and actively degrading, we still bridge immediately; synchronous voice is faster for genuine firefighting. But for the large middle tier of incidents that are real but not urgent, forcing everyone onto a call at 3am local time was pure cost with no speed benefit.
- Reserve synchronous bridges for incidents where speed of coordination outweighs the human cost of waking people.
- Structured async threads work if every message is self-contained — no "as I said above" references.
- A slow, linear degradation is a strong signal for a capacity or backlog problem, not a crash.
- Agree on the thread format before the incident, not during it.