I used to open five tabs and guess. Now I open one dashboard and let the data argue with my instincts, which turn out to be wrong more than I'd like.

I used to open five tabs and guess. Now I open one dashboard and let the data argue with my instincts, which turn out to be wrong more than I'd like.

My First Instinct Is Usually the Wrong Service

Early in my rotation, every page triggered the same reflex: check the service that paged me first, assume it's the root cause, start reading its logs. Six months and about thirty incidents later, I can say with confidence that the paging service is the root cause maybe a third of the time. The rest of the time it's a downstream symptom of something two or three hops away, and my gut consistently pointed at the wrong layer because it's the layer I know best, not the layer that's actually broken.

What Changed My Habits

The turning point was a dashboard-first habit: before touching any single service's logs, pull a cross-service view of error rate and latency for everything in the request path, sorted by which one deviated from baseline first:

from metrics
| where name in ("error_rate", "p99_duration")
| where service in ("api-gateway", "checkout-api", "inventory-svc", "payments-svc")
| summarize value by bin(1m), service
| sort by bin(1m) asc

Ordering by first deviation, not by which service's alert fired, has redirected me away from my first guess in roughly half of the incidents I've run this way. Payments looked broken because its error rate spiked, but the timeline showed inventory-svc's latency climbed ninety seconds earlier — payments was just the first thing to time out waiting on it.

Gut Feeling Still Has a Job

I don't think instinct is worthless. It's what tells me to check for a recent deploy, or that a graph "looks like" a memory leak shape I've seen before. But I've stopped letting it choose where I look first when the dashboard disagrees. The dashboard doesn't get tired at 4am and doesn't anchor on the service it happens to know best.

Instinct is a hypothesis generator, not a conclusion. Treat it that way and it stops costing you twenty minutes per incident.
  • Pull the cross-service timeline before diving into any single service's logs.
  • Sort by first deviation, not by which alert fired first — the paging service is often a downstream victim.
  • Use instinct to generate hypotheses, use the dashboard to rank and test them.
  • Keep a saved cross-service view for your critical path so it's one click, not five, at 4am.