We used to tell customers alert fatigue was a config problem they needed to fix. That was a convenient answer for us and not a true one.

For a long time, when a customer complained about alert fatigue, our support playbook pointed them at threshold tuning and deduplication settings. Technically correct advice, and also a way of putting the burden on the customer for a problem our defaults were making worse. We changed the defaults instead.

The default that was quietly causing the problem

New alert rules defaulted to a flat threshold with no built-in flapping protection — a metric oscillating just above and below a threshold could fire and resolve the same alert a dozen times in an hour, each one a separate page. Customers could turn on flapping protection, but it was opt-in, buried in advanced settings, and most people creating a rule in the moment of an incident never found it.

Making the safe thing the default

alert "high-error-rate" {
  when: error_rate > 0.05
  for: 3m           // must hold for 3 minutes, not one data point
  flap_protection: true   // default as of this release
}

The for clause and flap protection are both now on by default for every new alert rule, not opt-in settings. We measured a 41% reduction in total pages fired, at effectively unchanged incident detection time, across the cohort of workspaces that adopted the new defaults on existing rules.

What we owed customers who'd already tuned around the old default

Some customers had already hand-built their own flapping workarounds — duplicate rules with manual delays — to compensate for what should have been our job. We shipped a migration tool that detects those workaround patterns and offers to replace them with the native setting, rather than leaving two versions of the same fix stacked on top of each other.

Why we're publishing the number, warts and all

A 41% reduction in pages is a good headline, but we're deliberately also publishing the honest caveat: a small number of workspaces (around 4% in our rollout cohort) saw a slight increase in time-to-acknowledge on genuinely fast-onset incidents, because the three-minute hold delayed a page that, in hindsight, should have fired immediately. We're building a severity-aware override so critical-tagged alerts can opt out of the default hold, because a single default was never going to be correct for every incident shape.

  • Flap protection and minimum-duration hold now on by default for new alert rules
  • 41% fewer total pages fired with unchanged detection time in adopting workspaces
  • Migration tool detects and replaces manual flapping workarounds
  • Defaults changed instead of asking customers to tune around them