I've written alerts as an author for years. Only after a year of being woken up by other people's alerts did I understand what makes one good.

I've written alerts as an author for years. Only after a year of being woken up by other people's alerts did I understand what makes one good.

The Alert That Told Me Nothing

The one that changed my thinking was simple: "error rate high on api-gateway." Correct, technically. Useless at 3am. It told me something was wrong but nothing about where to look next, so every firing of that rule cost me the same ten minutes of triage regardless of how many times I'd seen it before, because it carried zero information beyond "look here."

Rewriting My Own Alerts With That Lesson

When I got back to writing rules for my own service, I started attaching the first diagnostic query directly in the alert payload, not just a link to a dashboard:

from traces
| where service == "query-planner"
| where span.attributes["error"] == true
| summarize count() by span.attributes["error.type"]
| sort by count() desc
| limit 5

Now the page itself shows the top error types by count before anyone has typed a single query. Half the time that's enough to identify the fix immediately; the other half it's still a massive head start over starting from a blank dashboard. The alert stopped being just a symptom and started being a diagnosis in progress.

The Other Thing I Changed: Saying What "High" Means

"Error rate high" doesn't tell you if that's 3% above a baseline of 0.1%, which is a real problem, or 3% above a baseline of 2.8%, which might be Tuesday. I started including the baseline and the current value directly in the alert title: "error rate 4.2% (baseline 0.3%)" versus "error rate 3.1% (baseline 2.9%)" reads completely differently to whoever's holding the pager, and lets them triage severity before opening a single tab.

An alert's job isn't to announce a problem. It's to save the responder the first five minutes of figuring out what you already knew when you wrote the rule.
  • Embed the first diagnostic query's output directly in the alert payload, not just a dashboard link.
  • State the baseline alongside the current value — a raw threshold number tells you nothing about severity.
  • Write every alert as if you'll be the exhausted person reading it at 3am, because eventually you will be.
  • An alert you'd resent receiving is an alert you should rewrite, not just leave for someone else to resent.