Half our fleet found the new load balancer instantly. The other half kept hammering an IP that had been decommissioned an hour earlier.

Half our fleet found the new load balancer instantly. The other half kept hammering an IP that had been decommissioned an hour earlier.

A Migration That Should Have Been Boring

We were replacing an aging load balancer with a new one behind the same hostname, a change we'd rehearsed in staging twice. The plan: update the DNS record, wait for propagation, decommission the old IP. In staging, propagation took under a minute. In production, three hours later, roughly a third of our ingest fleet was still resolving the old, now-dead IP and every request from those hosts was failing outright.

Splitting the Fleet by Behavior, Not Guessing

Rather than trying to reason about DNS caching in the abstract, we grouped failing requests by source host and looked for what separated the third that failed from the two-thirds that didn't:

from logs
| where message contains "connection refused" and service == "ingest-writer"
| summarize count() by host, resolved_ip
| sort by count() desc

Every failing host was resolving the same stale IP, and every one of those hosts had been running, unrestarted, for more than nine days — long enough for their local resolver cache to have picked up the record at a moment when a long-lived internal cache further upstream had cached it with a TTL of 6 hours instead of the 60 seconds the record was configured with. Someone had set that longer TTL eight months earlier during an unrelated incident to reduce DNS query volume on an overloaded internal resolver, then never reverted it once the resolver was fixed.

The Fix and the Longer Fix

The immediate fix was restarting the affected hosts to force a fresh resolution, which cleared the incident in about fifteen minutes once we knew what we were looking for. The three hours before that were spent because nobody suspected DNS caching at all — the mental model was "we updated the record, it should just work." We added a step to our load-balancer migration runbook to explicitly check the TTL on the record being changed and to keep the old IP alive for at least 2x the longest TTL found anywhere in the resolution chain, not just the value in the authoritative zone.

  • The TTL that matters is the longest one anywhere in the resolution chain, not the one in your authoritative zone.
  • Temporary TTL changes made during a past incident have a way of becoming permanent by omission.
  • Group failures by resolved IP, not just by symptom, when a migration seems to affect only part of the fleet.
  • Keep decommissioned infrastructure alive for a documented multiple of the worst-case TTL, not the best-case one.