My power went out mid-incident. The pager didn't care about my hurricane; it still expected an acknowledgment in ninety seconds.

My power went out mid-incident. The pager didn't care about my hurricane; it still expected an acknowledgment in ninety seconds.

The Single Point of Failure Was Me

I was primary on-call the week a tropical storm knocked out grid power across half my city. Around 9pm, our ingest cluster started rejecting writes — unrelated to the weather, just bad timing — and my apartment lost power fourteen minutes into the investigation. Laptop on battery, phone hotspot with maybe two bars, and a growing incident I could no longer meaningfully drive. I had built my entire on-call setup around the assumption that I, personally, would always have a stable connection. That assumption had never been tested until it mattered.

What Actually Kept the Incident Moving

What saved it wasn't heroics on my end, it was that our escalation policy had a real secondary, and our runbook was written well enough that she could pick it up cold:

from incidents
| where id == "INC-4471"
| project timestamp, actor, action
| sort by timestamp asc

The log showed the handoff took four minutes from my last update to her first action — not because the tooling was fast, but because the runbook had the exact HelixQL queries to check ingest backlog, the rollback command, and the names of the two people to page if it needed a second pair of hands. She didn't need me to explain the system; she needed the escalation policy to have a real second body who wasn't theoretical.

Redundancy Is a People Problem First

Afterward we audited every rotation for what we started calling "single-apartment risk": rotations where primary and secondary lived close enough together that one regional event could take out both. Two of our four rotations had that exact problem. We rebalanced schedules so primary and secondary are never in the same metro area during storm season, and we added a documented "I am about to lose connectivity, handing off now" protocol instead of expecting people to grind through degraded conditions out of guilt.

  • Your on-call redundancy plan is only as good as the geographic diversity behind it.
  • Runbooks that a secondary can execute cold, without a briefing call, are worth more than any dashboard.
  • Build an explicit "handing off due to personal outage" protocol so people disengage before they're forced to.
  • Audit rotations for shared local risk (weather, ISP, power grid), not just shared timezone.