DNS Failover: Why It's Slower Than It Looks

DNS controls which IP address a new lookup resolves to. It does nothing about a connection or session a client already established before the failure, and it can't force a resolver to honor a short TTL; some ISP and corporate resolvers cache well past whatever value you configured, regardless of what the record says.

Lowering TTL reduces propagation time for most resolvers that respect it, but some don't, and you generally have no visibility into which of your users are sitting behind one that ignores it. Plan around your worst observed resolver behavior, not your configured TTL value; the two can differ by a wide margin during a real incident.

A health check that only pings a load balancer, not the actual application behind it, can report healthy while the application itself is failing every real request. Point health checks at something that reflects genuine application health, and set a sensible interval and failure threshold so a single transient blip doesn't trigger a failover nobody actually needed.

Run the check from more than one vantage point if your provider supports it. A health check running from a single location can itself lose connectivity to a perfectly healthy region, triggering a failover based on a problem with the checker rather than the service being checked, which is its own kind of false positive worth designing around.

Anycast routing announces the same IP address from multiple locations and lets network-layer routing send each client to the nearest healthy one, sidestepping the DNS caching problem entirely because no new lookup is required to shift traffic. The tradeoff is that it needs a network provider that actually supports it and represents a bigger infrastructure commitment than DNS-based failover.

Most small and mid-sized teams start with DNS failover, since it's simpler to set up and reason about, and only move to anycast once DNS's propagation delay has caused a real incident that a lower TTL alone couldn't fix.

The two approaches aren't mutually exclusive. A common middle ground is anycast for the entry point that serves traffic day to day, with DNS-based failover kept as a fallback path for the rare case that even the anycast network itself needs to route around a whole provider's regional outage.

• Set TTL as low as your DNS provider allows for records you might need to fail over quickly, while accepting that some resolvers still won't respect it.
• Point health checks at something that reflects real application health, not just infrastructure being reachable.
• Test an actual failover on a schedule, not only once at initial setup, since DNS and health check behavior can drift as infrastructure changes.
• Know your real observed failover time from an actual test, not the number implied by your TTL setting.

Even a fast, well-tested failover leaves a window where some share of users are still resolving the old address, and their requests either time out or hit a region that's actively being drained. Design the client side for that gap: reasonable connection timeouts instead of ones long enough to leave a user staring at a spinner, and retry logic on the client that doesn't just hammer the same failed address in a tight loop.

Communicate the realistic gap internally too, to whoever owns customer-facing status updates. Telling customers a failover completed the moment DNS changed, when a meaningful share of them are still resolving the old address for several more minutes, sets an expectation the actual system can't meet.