A dozen services should not all page the same catch-all channel. Route by ownership metadata baked into the alert rule itself, not a manual triage step.

The default failure mode for a growing alert set is everything landing in one channel, followed by a human deciding who should actually own each page. That triage step is slow exactly when speed matters most. Routing on ownership metadata at alert-creation time removes it entirely.

1. Tag ownership at the source

Ownership belongs on the service, not on the alert rule, so it stays correct even as alert rules are added and removed. Set it as a resource attribute at SDK initialization, the same place service name and environment are already set.

require('@helix-obs/sdk').start({
  service: 'checkout-api',
  resourceAttributes: {
    'team.owner': 'payments',
    'team.pagerduty_service': 'PD-CHECKOUT-01',
  },
});

2. Reference the attribute in the alert notify clause

Instead of hard-coding a destination in every alert rule, resolve it dynamically from the same attribute set on the traced service, so a new alert on an existing service routes correctly without any extra configuration.

ALERT high_error_rate
FROM (
  from traces
  | where team.owner == "$owner"
  | summarize countif(status_code >= 500) * 1.0 / count() by service.name
)
WHEN value > 0.05
NOTIFY pagerduty:${team.pagerduty_service}

3. Fall back to a default owner, never to nothing

Services occasionally ship without the ownership attribute set, usually a new one still under active development. Configure a fallback destination for anything missing the tag, so a gap in metadata produces a page to a platform team rather than an alert that silently has nowhere to go.

4. Audit ownership coverage on a schedule

Ownership tags drift as teams reorganize and services change hands. A monthly query listing every service with either a missing owner tag or an owner tag pointing at a paging integration that no longer exists catches the gap before an incident does.

from traces
| summarize count() by service.name, team.owner
| where isnull(team.owner)

This audit is worth running as a scheduled job rather than a manual quarterly task, since the cost of finding a stale route during an actual incident is far higher than the cost of a query that runs itself every month. A service silently paging a defunct integration looks identical to a healthy alert setup right up until it is needed.

  • Set ownership as a resource attribute on the service, not inside each alert rule.
  • Resolve the paging destination dynamically from that attribute in the notify clause.
  • Always define a fallback route for services missing the ownership tag.
  • Audit ownership coverage on a monthly schedule, tags drift as teams reorganize.