An internal service-to-service TLS certificate expired quietly, and the alert meant to catch it 30 days early had been silently failing for two months.

At 17:00 on March 11th, the mTLS certificate used between our ingest gateway and the storage tier expired. Every write to Helix's storage layer began failing the handshake within seconds. The certificate-expiry alert that should have paged us a month earlier had been broken since January.

Timeline

17:00:00 — the internal CA-issued certificate for ingest-gateway reaches its expiry timestamp exactly. 17:00:02 — new TLS handshakes to the storage tier start failing with a certificate-expired error; existing long-lived connections keep working briefly. 17:03:00 — as connections cycle, write success rate starts dropping; by 17:06:00 it's under 40%. 17:06:30 — on-call paged on write error rate. 17:09:00 — responder checks certificate expiry directly (not from the alert, which never fired) and finds it lapsed six minutes earlier. 17:12:00 — an emergency short-lived certificate is issued and rolled out via the existing rotation pipeline, manually triggered. 17:19:00 — write success rate back to 100%.

Root cause

Two things stacked: the certificate rotation automation had a bug introduced in a January refactor that made it silently skip certificates tagged with a particular internal service class, including this one. Separately, the "certificate expires in 30 days" alert queried a metrics export that had also broken in that same refactor, so the one signal that should have caught the first bug was itself broken.

The fix

Once we knew to look, the query that should have paged us in February was straightforward:

from metrics
| where metric.name == "tls_cert_expiry_seconds"
| summarize min(value) by service
| where value < 30 * 24 * 3600

We fixed the rotation automation's service-class filter bug, and separately fixed the metrics export. Critically, we also added a second, independent expiry check that doesn't share any code path with the rotation system — an external probe that connects to each internal endpoint and reads the certificate directly.

What changed

We no longer treat "the alert didn't fire" as evidence nothing is wrong. The new external certificate probe is deliberately decoupled from the rotation and metrics pipelines so a bug in one can't silently disable detection in the other. We also added a quarterly manual audit of all certificate expiry dates as a belt-and-suspenders check.

  • Certificate expiry monitoring must be independent of the rotation system it's watching.
  • An alert that silently stops firing is worse than no alert — test alert pipelines themselves, not just the conditions they check.
  • Add an external, direct-connection probe for anything TLS-related; don't trust only internal metrics.
  • Quarterly manual audits catch what automation regressions miss.