Most routing engines can look at a severity label and a service name. Ours can now look at anything in the signal that triggered the alert.

The old Helix router matched on three fields: severity, team, and service. That was enough until a customer asked us why a database connection-pool exhaustion alert and a single flaky health check both paged the same on-call engineer at 3am with the same urgency.

Severity labels are not a routing strategy

A label is a static guess made when the alert rule was written. It can't know that this particular spike is affecting 40% of your fleet versus one canary pod. We wanted routing decisions made from the actual data in the alert event, not a static tag someone set six months ago.

Routing rules as HelixQL

Routing rules now run against the alert's underlying signal at fire time:

route "db-pool-exhaustion" {
  when: alert.name == "pool_exhaustion" and affected_hosts_pct > 25
  to: pagerduty("database-oncall"), slack("#db-incidents")
  else_to: slack("#db-watch")
}

The else_to branch matters as much as the primary one: a rule that always pages is a rule your team will eventually mute. Now the same alert definition can page for a fleet-wide event and just post to Slack for an isolated blip, without maintaining two separate alert rules.

What this replaced

Before this shipped, teams worked around the limitation by writing near-duplicate alert rules with different thresholds feeding different routes — one customer had 340 alert rules, of which we later found 90 were severity-tier duplicates of another rule. After migrating to conditional routing, they got that down to 210 with no coverage loss.

Testing a routing rule before it pages anyone

The riskiest part of shipping this was letting a routing rule reference live data without a way to check what it would have done historically. We added a simulate mode that replays a proposed rule against the last 30 days of fired alerts and reports where it would have routed each one, so a team can catch a rule that accidentally routes everything to the same quiet Slack channel before it goes live, not after an incident nobody got paged for.

  • Routing rules evaluated against real alert payload data, not just static labels
  • Conditional else_to branches to avoid duplicate alert rules
  • Routes to PagerDuty, Opsgenie, Slack, and webhook targets from one rule
  • Full routing decision trace attached to every fired alert for audit