Deleting an alert feels riskier than adding one, so most teams never do it. A small, repeatable ritual made it routine instead of scary.

Nobody wants to be the person who deleted the alert that would have caught the next big outage. That fear is why alert graveyards never actually get cleaned up — until we made retirement a ritual instead of a judgment call.

Demote before you delete, always

No alert goes straight from paging to gone. It first moves to a silent, logged-only state for two weeks, so if it would have fired during that window, we know exactly what we'd have missed before committing to the deletion. This single step is what made engineers comfortable retiring alerts they'd been afraid to touch for years.

from alert_definitions
| where status == "demoted" and demoted_at < ago(14d)
| join kind=leftouter (alert_history | where fired_at > ago(14d)) on alert_name
| summarize would_have_fired = count() by alert_name
| where would_have_fired == 0

Require a one-line reason, logged permanently

Every retired alert keeps a permanent, searchable record of why it was retired and who approved it, even after the alert definition itself is gone. This isn't bureaucracy for its own sake — six months later, when someone asks "didn't we used to alert on this?", the answer is a two-second lookup instead of an afternoon of git archaeology.

Celebrate the deletion, not just the addition

We started including alert retirements in the same weekly update where we announce new dashboards and features, treating a well-justified deletion as a genuine improvement rather than something to quietly do and never mention. That framing shift did more to unstick the backlog of zombie alerts than any process change.

Make the first retirement someone else's, not your own

The very first alert we ever retired under this ritual belonged to the engineer proposing the process, specifically so nobody could accuse the ritual of being a way to dodge accountability for someone else's alert. Leading with your own noisy alert, in public, did more to make the process feel fair than any amount of explaining the safeguards would have.

  • Demote to silent, logged-only for two weeks before permanently deleting any alert.
  • Confirm nothing would have fired during the silent period before committing.
  • Keep a permanent, searchable record of why every alert was retired.
  • Treat alert retirement as a visible win, not a quiet, risky chore.
  • Have the process's proposer retire their own alert first to build trust in it.