A rolling deploy overlapped with a leader-election handoff just long enough for two pods to both believe they were the leader, and both ran the nightly billing job.

On the night of February 17th, our nightly billing-reconciliation job ran twice, 40 seconds apart, from two different pods that each believed they held the leader lock. Duplicate charge-adjustment records were created for roughly 1,100 accounts before a downstream idempotency check caught the second run mid-way.

Timeline

01:00:00 — a rolling deploy of billing-scheduler begins, replacing pods one at a time. 01:00:12 — the pod currently holding the leader lease (a 15-second TTL lease renewed every 5 seconds) is sent SIGTERM as part of the rollout, with a 10-second grace period. 01:00:14 — a new pod starts and, seeing the lease about to expire, acquires leadership. 01:00:16 — the old pod, still finishing its grace period, renews its lease one more time before shutting down, briefly re-asserting leadership. 01:00:18 — both pods believe they are leader and both trigger the 01:00 cron job. 01:00:19–01:03:40 — both job instances run concurrently against the billing database. 01:03:41 — an idempotency check on the adjustment-write path (keyed on account_id + billing_period) starts rejecting the second run's writes as duplicates, but not before ~1,100 accounts were double-processed in the window before the check caught up.

Root cause

The leader-election library allowed a lease renewal to succeed even after the holder had received a termination signal, because the renewal check didn't consult shutdown state. Combined with a rollout that didn't drain the leader role before sending SIGTERM, this created a real window where two pods both held a valid lease.

The fix

We found the double-run using a HelixQL query joining job-start events by job name within a tight time window:

from logs
| where message == "job.started" and job.name == "billing_reconciliation"
| summarize starts=count(), pods=dcount(pod.name) by bin(time, 5m)
| where starts > 1

We fixed the immediate bug by making the deploy process explicitly release the leader lease on SIGTERM rather than passively expiring it, and added a distributed lock (not just a lease check) around the job's critical section as defense in depth.

What changed

Every cron-style job in the fleet now has an idempotency key check as a hard requirement, not an optional safety net — this incident validated that it works, but it shouldn't be the only thing standing between us and duplicate billing writes. We also added a pre-deploy check that no rollout may proceed while a leader handoff is in progress.

  • Release leadership explicitly on graceful shutdown; don't rely on lease expiry alone.
  • Every scheduled job needs an idempotency key on its side effects, no exceptions.
  • Block rollouts from starting mid leader-handoff.
  • Alert on duplicate job-start events, not just job failures.