Setting Up DNS Failover That Actually Fails Over When It Matters

A DNS record's time to live determines how long a resolver, and any client caching that resolver's answer, will keep using a stale address after a failover has already happened at the DNS layer. A high TTL set for caching efficiency during normal operation directly works against fast recovery during an incident, since clients keep hitting the failed endpoint until their cached answer expires.

Lower the TTL on records you expect to use for failover, well before you actually need it, since a TTL change itself takes time to propagate. Making that change during an active incident is too late to help with the incident already happening; it only helps the next one.

Resolvers along the chain between your DNS provider and the client can cache more aggressively than your configured TTL suggests, and some client libraries or operating systems cache DNS answers independently of what the resolver returns, sometimes ignoring TTL guidance from the record entirely. This is why a failover that works cleanly in a controlled test can behave inconsistently across real client environments during an actual incident.

Anycast routing, where the same IP address is announced from multiple physical locations and network routing picks the nearest one, avoids the DNS caching problem for the failover decision itself, since the routing change happens at the network layer rather than depending on a client re resolving a name. It's a meaningfully more reliable mechanism for failover, at the cost of more complex infrastructure to operate.

A health check that only confirms a server responds to a ping tests a much shallower layer than the actual failure modes that matter, a database connection pool exhausted, a downstream dependency timing out, a process technically running but unable to serve real traffic correctly. Health checks feeding a failover decision need to test something closer to a real request, not just basic reachability.

Set the check interval and failure threshold deliberately: too sensitive and a brief blip triggers an unnecessary failover, which has its own cost and risk; too lenient and a real outage runs longer than it needs to before failover kicks in. Base the threshold on your actual historical false positive rate from monitoring, not a guess.

A health check feeding a failover decision should cover:

• An exhausted database connection pool, not just a server that still answers a ping.
• A downstream dependency that is timing out while the process itself keeps responding.
• A process that is technically running but unable to serve real traffic correctly.
• Behavior close to a real request, rather than only confirming that the host is reachable.

Run a deliberate, scheduled failover test in a controlled window, actually taking the primary endpoint down and confirming traffic reroutes within the time your policy claims it will, rather than trusting the configuration was correct the day it was set up. This is the only way to catch the caching and propagation issues described above before they show up during a real incident.

Measure actual time to recovery during that test, not just whether failover eventually happened, and compare it against what your team assumes the recovery time is. A gap between assumed and actual recovery time is worth fixing before it becomes part of an incident postmortem instead of a routine test result.

During an actual outage is a bad time for an on call engineer to be discovering how failover is supposed to work for the first time. Write down which records have failover configured, what the expected recovery time is based on your last real test, and what manual steps, if any, are needed if the automatic mechanism doesn't behave as expected.

Keep that documentation next to your other incident runbooks, not in a separate infrastructure wiki nobody checks during a page. The value of a well tested failover policy is largely lost if the person responding to an incident doesn't know it exists or doesn't trust it enough to wait for it to work.