Teams reach for higher thresholds when they are drowning in pages. Often the real fix is not fewer alerts — it is sending each one to the person who can actually act on it.
The noise is usually misrouted
When every alert lands in one channel, everything competes for attention and nothing gets it. The database alert wakes the frontend engineer, who mutes the channel, and now the real fire is muted too. Routing is triage, and triage is the difference between signal and noise.
Route on the data, not on the dashboard
Because HelixQL alerts carry the labels of the data that fired them, you can route on ownership directly:
alert "high error rate"
| when rate(errors) by service > 0.05 for 5m
| route to team_of(service)
| severity if rate(errors) > 0.2 then "page" else "ticket"Two tiers, not one
- Page for things a human must act on now — customer-facing and getting worse.
- Ticket for things that need attention but not adrenaline — degraded, not down.
The mistake is treating every alert as a page. Most are tickets that got promoted by fear.
Close the loop
Every page that turns out to be non-actionable is a routing bug. We review them weekly and either re-route, downgrade, or delete. The alerting system, like any other, needs maintenance — and its bug tracker is your on-call log.