DNS Failover, Answered: TTLs, Health Checks and What Actually Fails Over

A TTL is a request to resolvers to stop caching a record after a certain time, not a guarantee. Some ISPs and corporate resolvers cache longer than the TTL specifies, and some clients or applications cache a resolved IP independently of DNS entirely, holding a connection open to the old address well past when DNS itself would have updated. A low TTL shortens the tail of stale traffic after a failover; it doesn't eliminate it, and treating it as instant is the gap that shows up mid-incident when a meaningful slice of traffic is still hitting the dead endpoint minutes after the DNS record changed.

For example, imagine a team sets a low TTL, triggers failover and sees the dashboard turn green within a minute. Meanwhile, request logs at the old address show traffic still arriving from resolvers and long-lived client connections that never re-resolved. Build that tail into your incident runbook. Tell responders that failover completes gradually, and don't declare recovery until traffic to the dead endpoint has actually dropped. Watching the old endpoint's request volume is a more reliable signal than checking that the DNS record changed.

Typically, a TCP connection or an HTTP request against a specific endpoint, checked for a response code or a simple string match, nothing more. That confirms the process is up and the port is listening; it doesn't confirm the actual request path behind that endpoint is healthy. A health check hitting a lightweight status page can report green while the real application, several dependencies deep, is failing every genuine request. Point the health check at something that exercises a real, representative path, not the cheapest endpoint to keep green.

Anycast routes a request to the nearest healthy instance at the network layer itself, so many regional outages resolve automatically without any DNS record ever changing, which is a meaningfully faster response than DNS propagation allows. It only helps if you're actually running instances in more than one location for it to route between; anycast in front of a single region doesn't add any failover capability, it just adds a routing layer with nothing to fail over to.

The primary region looking healthy to the health checker while it's actually serving errors to real users, because the health-check endpoint doesn't touch the same dependency that's actually broken. A database connection pool exhausted for real user traffic but not for the lightweight health-check query, or a degraded third-party dependency that only a subset of real endpoints call, both produce this exact gap. The health check needs to fail when the thing users experience fails, which usually means it has to be less lightweight than the endpoint that's cheapest to keep monitoring, even though that makes the check itself a little more expensive to run on every interval.

Run a scheduled, deliberate failover drill, not just a one-time test at setup. Fail over to the secondary on a known schedule, confirm real traffic actually shifts within the time you expect, and fail back. A failover path that hasn't been exercised in months is effectively untested, because DNS provider configuration, health check endpoints, and the secondary region's actual capacity to handle full production load can all drift silently in the meantime, and the first time anyone would notice is during the incident the failover was built for.

Run a failover drill using these steps:

1. Fail over to the secondary on a known schedule, instead of testing once at setup and trusting it for years.
2. Confirm real traffic actually shifts within the time you expect, allowing for resolvers and clients that cache longer than the TTL.
3. Push enough real traffic through the secondary to see how it behaves at production load, since standby sizing is a common failure.
4. Fail back, and note any drift in DNS provider configuration or health check endpoints before the next drill.

The most common finding from a real drill isn't in the DNS layer at all: it's a secondary region sized for standby, not full load, that starts failing the moment real traffic actually lands on it. A drill that only confirms DNS resolved correctly, without pushing enough real traffic through the secondary to see how it behaves under production load, misses exactly the failure a real incident would expose. Size the secondary for what it would actually need to carry, and confirm that with traffic, not with a capacity spreadsheet.