I was sure it was the new deploy. The dashboard kept insisting otherwise. I spent forty minutes proving myself wrong before I finally listened.

I was sure it was the new deploy. The dashboard kept insisting otherwise. I spent forty minutes proving myself wrong before I finally listened.

A Deploy Fifteen Minutes Before the Alert

Timing is a hell of a drug. A deploy to our notification service went out at 2:47pm. The p99 latency alert fired at 3:02pm. Fifteen minutes felt close enough to be causal, and I spent the first half hour of the incident almost entirely focused on that deploy — diffing the change, checking for anything that could explain a latency regression, getting a rollback staged and ready to go.

What the Cross-Service View Actually Showed

The rollback was staged, but before pulling the trigger I ran the same cross-service baseline comparison I've learned to run before acting on instinct:

from metrics
| where name == "p99_duration"
| where service in ("notification-svc", "template-render", "asset-cdn")
| summarize value by bin(2m), service
| sort by bin(2m) asc

notification-svc's latency had actually started climbing at 2:31pm — sixteen minutes before its own deploy went out, and template-render's latency climbed in near-lockstep with it. The deploy I was so sure about happened to land in the middle of an unrelated, already-in-progress degradation in a shared dependency. Rolling it back would have done nothing and cost us the twenty minutes it takes to verify a rollback actually improved things, time we didn't have.

Following the Real Timeline

template-render's climb traced to asset-cdn, whose cache hit ratio had dropped after an upstream CDN configuration change unrelated to anything on our side, pushing more requests through to origin and slowing template rendering, which slowed notifications. The actual fix was on the CDN config, not our deploy at all. I'd been anchored on the deploy because it was recent and visible, not because the evidence pointed there — the evidence, once I actually looked at it against a real timeline instead of a hunch, pointed sixteen minutes earlier and one service over.

Recency and visibility make a deploy feel causal. Only a timeline comparison tells you if it actually is.
  • A deploy shortly before an alert is a hypothesis, not a conclusion — check the actual onset time against it.
  • Cross-service baseline comparisons can rule out your leading theory faster than digging deeper into it.
  • Don't stage a rollback as a substitute for confirming causality; it costs real time if it's the wrong move.
  • The most recent visible change is not always the most likely cause — it's just the easiest one to blame.