Our incident notes from that night were thin, written in the middle of the fire. Weeks later, we rebuilt the real timeline entirely from telemetry.

Our incident notes from that night were thin, written in the middle of the fire. Weeks later, we rebuilt the real timeline entirely from telemetry.

The Notes We Had Weren't Enough

The incident channel from that night had maybe fifteen messages across ninety minutes, several of them just "still looking" or a link to a graph with no caption. Understandable in the moment — nobody's writing a polished narrative while things are on fire — but it left us with a postmortem draft that was mostly guesswork about the order things actually happened in.

Rebuilding Order From the Data Itself

Rather than rely on memory, we queried every signal touching the affected services across the incident window and stitched them into a single chronological view:

from traces, logs, metrics
| where service in ("auth-service", "session-store", "api-gateway")
| where timestamp between ("2026-06-02 21:40:00", "2026-06-02 23:10:00")
| union
| sort by timestamp asc
| project timestamp, source, service, summary

The reconstructed timeline showed something the human notes had gotten subtly wrong: we'd remembered session-store's errors as the first sign of trouble, but the unified timeline showed auth-service's latency had already been climbing for six minutes before session-store showed a single error. The chat log's first mention was of session-store simply because that's what triggered the page — the actual onset was earlier and in a different service, and no human had noticed it live because nobody was watching auth-service's dashboard until it was already implicated.

What Changed Because of the Rebuild

The corrected timeline changed our root cause finding entirely — from "session-store bug" to "auth-service latency causing session-store timeouts, itself caused by a slow token-refresh path we hadn't examined at all in the original response." We now treat the incident channel transcript as a starting hypothesis, not ground truth, and rebuild the real timeline from raw telemetry for anything above Sev-2 before finalizing a postmortem, specifically because human memory during an active incident optimizes for fighting the fire, not for accurately recording its shape.

  • Incident chat logs record what got noticed, not necessarily what happened first — the two can differ by minutes that matter.
  • Rebuild a unified cross-signal timeline from raw telemetry before finalizing a postmortem's root cause.
  • The service that triggered the page is not automatically the service where the problem started.
  • Treat live incident notes as a hypothesis to verify afterward, not as the final record.