A DNS cutover during a data-center migration should have been invisible. A too-long TTL and a stale resolver cache made it a 20-minute regional blackout.

On March 4th, we cut DNS over to a new load balancer IP as the final step of a data-center migration. The TTL on that record was still set to 3600 seconds from months earlier. For 20 minutes, a meaningful slice of traffic kept resolving to an IP that no longer answered.

Timeline

15:00:00 — the old load balancer is decommissioned as planned, on schedule, after the migration runbook's traffic-drain step. 15:00:05 — DNS is updated to point at the new load balancer, but the existing record's TTL of 3600s means resolvers that cached the old answer any time in the last hour won't re-query until their individual TTLs expire. 15:00:10 — requests from clients with a fresh cache resolve correctly and work fine. 15:00:10–15:20:00 — requests from clients with a stale cache continue hitting the decommissioned IP and time out. 15:03:00 — error rate alert fires for elevated connection timeouts from a subset of regions. 15:07:00 — on-call correlates the errors to specific client IP ranges via HelixQL and identifies stale DNS as the pattern, not a backend issue. 15:09:00 — team stands up the old load balancer's IP as a temporary passthrough to the new one, mitigating for stragglers. 15:20:00 — the last observed stale-cache traffic clears as TTLs finish expiring.

Root cause

The migration runbook had a step to lower the DNS TTL to 60 seconds 24 hours before cutover — standard practice — but it was skipped because the runbook's DNS section was written for a different record than the one actually being cut over that day.

The fix

We spotted the pattern using a query that grouped failing connections by resolved IP rather than by error type:

from logs
| where message == "connection.timeout" and time > ago(30m)
| summarize count() by dest_ip, client.region
| sort by count desc

Every failing connection pointed at the decommissioned IP, and it was concentrated in three regions — a strong signal for DNS caching behavior rather than a real backend fault. Standing up a passthrough on the old IP was faster than waiting out every possible resolver's TTL.

What changed

The migration runbook template now has a mandatory pre-cutover step that verifies the actual TTL on the actual record being changed, not a generic reminder. We also now keep decommissioned load balancer IPs alive as simple passthroughs for 2x the old TTL value as standard practice, so a missed TTL drop degrades gracefully instead of blackholing traffic.

  • Verify the TTL on the specific record being cut over — don't trust a generic runbook step.
  • Keep decommissioned IPs alive as passthroughs for 2x the old TTL as a safety margin.
  • Group failure analysis by destination IP early — it surfaces DNS-class issues fast.
  • Treat DNS TTL reduction as a scheduled, tracked pre-migration task with its own verification.