DNS Failover: What Actually Happens When a Region Dies

A DNS provider that monitors an endpoint's health and swaps the returned IP when it fails is straightforward to set up and understood by most teams. Its ceiling is TTL: a low TTL means faster failover propagation but more DNS query volume and less benefit from caching; a high TTL means resolvers and clients hold onto the old, dead IP longer after the health check has already failed over. Many clients and some resolvers also ignore TTL entirely and cache longer than instructed, which means even a well-tuned TTL doesn't guarantee every client fails over on your expected timeline.

Anycast routes traffic to the topologically nearest healthy instance advertising the same IP address at the network layer, which means failover doesn't depend on DNS propagation or client caching behavior at all, a dead region simply stops being routed to by the network. This is a heavier infrastructure lift, it typically requires your own IP address space and BGP relationships or a provider that offers anycast as a managed service, and it's usually the right investment once DNS TTL-bound failover has proven too slow or unreliable for your actual recovery time requirements.

Health-check failover and anycast both assume your DNS provider itself stays up; a DNS provider outage, which has happened to every major provider at some point, takes down failover along with everything else if you're single-sourced. Running authoritative DNS across two providers, with your zone kept in sync between them, protects against this specific failure mode, though it adds operational complexity, particularly around keeping records consistent and coordinating any changes across both providers without drift.

Run an actual failover drill in a lower environment: take the primary region offline the way a real outage would, and time how long it actually takes for traffic to shift, including the client-side caching behavior you can't fully control from your DNS configuration alone. This regularly surfaces gaps a written runbook doesn't: a health check with too generous a failure threshold, a CDN layer caching DNS resolution longer than the origin's TTL, a client library that caches DNS results for its own process lifetime regardless of TTL.

A useful failover drill covers these points:

• Take the primary region offline in a lower environment the way a real outage would.
• Time how long traffic actually takes to shift, including client and CDN caching you can't fully control.
• Test internal resolution too, since split-horizon DNS can still point at the dead region.
• Compare the measured recovery time against your availability target, then tune TTL and health checks.
• Update the status page and incident channels even when the failover is clean.

The gap between a 99.9% and a 99.99% availability target isn't a rounding error, it's the difference between roughly nine hours and roughly one hour of allowed downtime across a full year¹. A health-check interval and failure threshold tuned for a looser target will burn through a tighter target's entire annual downtime budget in a single slow-to-detect incident, so pick your TTL and health-check cadence starting from the actual availability target you've committed to, not a default your DNS provider ships with.

If internal services resolve your domain differently than external clients do, through split-horizon DNS or an internal resolver override, a failover drill that only tests the external path can pass while an internal dependency still points at the dead region. Map out every place your domain resolves differently for internal versus external traffic before you trust a single drill's result as proof the whole system fails over correctly, since it's common for an internal service's DNS configuration to drift out of sync with the external-facing failover setup without anyone noticing until it matters.

A successful automated failover still means your primary region is down, and status page updates, internal incident channels, and any customer-facing communication should reflect that even if end users never noticed a service interruption. Treat a clean failover as a real incident worth a lightweight postmortem, not just a success story, since the same health check and threshold tuning that made the failover fast this time is the thing most likely to have drifted by the next time it's actually needed.