Our first draft of this retro blamed the wrong service. A clearer timeline, written before the fix instead of after, caught the mistake before it shipped.

On May 22nd, checkout latency spiked and the first responder's working theory — a bad deploy to pricing-service — was wrong. We almost wrote that up as the root cause. Rewriting the timeline in strict chronological order, before drafting any conclusion, is what caught the error.

Timeline

16:00:00 — pricing-service deploys v5.3.0, a config-only change. 16:04:00 — checkout p95 latency begins climbing from 190ms toward 900ms over the next ten minutes. 16:08:00 — on-call, seeing the deploy four minutes prior, opens the incident assuming pricing-service v5.3.0 as the cause and begins preparing a rollback. 16:11:00 — before rolling back, a second responder insists on building the full timeline first, per the postmortem process, and pulls every deploy and config change across all services touched by the checkout path in the preceding 30 minutes. 16:14:00 — that review surfaces a second, less obvious change: a feature flag toggled at 16:03:30 on inventory-service, one minute before pricing-service's deploy, that enabled a new stock-check call on the checkout path for 100% of traffic. 16:17:00 — the flag is identified as the actual correlated cause via HelixQL; pricing-service's deploy is unrelated. 16:19:00 — flag disabled; latency recovers within 90 seconds.

Root cause

The new stock-check call added a synchronous dependency to a service with its own 400ms p95, adding directly to checkout's critical path. It had nothing to do with pricing-service — the two changes just happened 90 seconds apart, and the more visible one (a deploy, versus a flag toggle) was the natural first suspect.

The fix

The query that separated correlation from coincidence:

from traces
| where service == "checkout-api" and span.name == "checkout.total"
| summarize p95=p95(duration) by bin(time, 1m)
| join (from audit_log | where time between (ago(30m), now())) on 1==1

Joining the latency trend against every change in the window, not just the most recent one, is what surfaced the flag toggle. Had we rolled back pricing-service first, the timeline would have "confirmed" the wrong theory purely by coincidence — the flag might have still been live, or might have been caught later by someone else, at greater cost.

What changed

Our postmortem template now requires the full timeline of every change (deploys, flags, config, infra) in the relevant window to be built before any root-cause section is drafted, specifically to prevent anchoring on the first plausible theory. We consider this one of our most valuable process fixes this year, even though the "incident" itself was fairly minor.

  • Build the complete change timeline — deploys, flags, config, infra — before drafting a root-cause theory.
  • Don't let "most recent deploy" default to "most likely cause"; verify with correlated data.
  • Feature flag toggles deserve the same timeline visibility as deploys — they're not lower-risk just because they're reversible.
  • A wrong rollback has a cost even when the outage still resolves — it burns time and trains bad instincts.