A latency alert that pages on a single noisy spike is worse than no alert at all. Here is how to build one that survives real traffic.

The first alert most people write fires too often, gets muted within a week, and teaches the whole team to ignore Helix notifications. Building it correctly the first time takes five extra minutes and saves months of alert fatigue.

1. Start from a query, not a threshold

An alert rule in Helix is a HelixQL query plus a condition. Write the query first and run it against the last seven days of data before attaching any threshold, so the number chosen reflects actual traffic rather than a guess.

from traces
| where service.name == "checkout-api"
| summarize p95(duration) by bin(timestamp, 5m)

2. Pick a window wider than the noisiest normal spike

Look at the seven-day chart from step one. If p95 latency naturally touches 800ms during the Monday morning traffic ramp, a threshold of 700ms sustained for one evaluation window will page every Monday regardless of whether anything is actually wrong. Set the threshold above the noisy-but-normal ceiling, and require it to hold for multiple consecutive windows.

ALERT checkout_api_p95_latency
FROM (
  from traces
  | where service.name == "checkout-api"
  | summarize p95(duration) by bin(timestamp, 5m)
)
WHEN p95(duration) > 900ms FOR 3 consecutive windows
NOTIFY pagerduty:checkout-oncall

3. Route it to a person, not a channel

Alerts that land in a Slack channel get acknowledged by whoever happens to be looking at the time, which in practice means sometimes nobody. Route the first alert through a paging integration with an explicit on-call schedule attached, even for a low-severity rule, so there is always exactly one owner per page.

4. Test it against history before it goes live

Every alert rule in Helix has a Backtest tab. Run the new rule against the last thirty days before enabling it, and read the list of times it would have fired. If that list includes ordinary Monday mornings or the weekly batch job window, the threshold is still wrong, tune it before anyone gets paged for it.

  • Write and inspect the underlying query against real history before attaching a threshold.
  • Set thresholds above the noisy-but-normal ceiling, and require sustained breach, not a single sample.
  • Route every alert to a named on-call schedule, not a channel.
  • Backtest against thirty days of history before enabling the rule.