Multi-Region Routing Is Easy Until a Region Actually Fails

DNS-based or anycast routing that sends users to their geographically nearest region is a performance optimization, not a resilience strategy on its own. It reduces latency for the common case and does nothing by itself to redirect traffic away from a region that's actually failing.

Failover needs active health checks that detect a region's degradation, not just its complete unreachability, and routing logic that reacts to those checks within seconds, not the minutes a DNS TTL alone might take to propagate. These are two separate systems that need to work together, not one system doing double duty.

A region can be up and ready to serve traffic while its copy of the data is seconds or minutes behind the failed region's last known state, depending on your replication strategy. Failing over instantly to a lagging replica means some recent writes are temporarily invisible, or in a worse case, get overwritten once the original region recovers and reconciliation runs.

Be explicit about what your replication actually guarantees during failover: how much data might be stale or at risk, and what your reconciliation process does when the failed region comes back with writes the other region never saw. An honest answer here shapes what you can actually promise users during a regional incident.

A failover runbook that's never been executed against real traffic has unknown gaps, the same way any other untested runbook does. Scheduling an actual failover drill, ideally against a small percentage of real traffic during low-risk hours, surfaces problems a tabletop review never will: a hardcoded region reference in a config file, a dependency that only exists in one region, a certificate that wasn't provisioned in the failover target.

This is uncomfortable to schedule and consistently worth it. Teams that have actually drilled a regional failover recover meaningfully faster during a real one than teams executing the process for the first time under pressure.

For example, a team planning its first drill can start by listing every component that must run in the failover region: the main application, the background job scheduler, certificates, configuration files, and any internal tooling. It then sends a small share of real traffic to the target region during a low-risk window and checks each component against the list. Anything missing becomes a fix before the next drill. A useful decision rule: count a drill as passed only when the supporting infrastructure, not just the user-facing path, works in the target region, and record how long recovery actually took.

Say a company runs active-active across two regions for its main application but discovers, during its first real failover drill, that a background job scheduler was only ever deployed in the primary region. Failing over the main application worked cleanly; the scheduled jobs, invoices, reports, reminders, silently stopped running until someone noticed hours later.

This is a common pattern: the user-facing path gets the multi-region investment, while supporting infrastructure quietly remains a single point of failure nobody flagged. A drill is what surfaces this kind of gap before a real regional outage does, when the missing jobs are a lot harder to notice quickly.

• Latency-based routing treated as if it were also failover, with no active health-based redirect
• Replication lag and reconciliation behavior never explicitly documented or tested
• Background jobs, cron tasks, or internal tooling deployed in only one region
• Failover never drilled against real traffic, only reviewed as a document

Multi-region architecture is expensive to build and maintain well, and the investment should match a deliberately chosen availability target, not an assumed one. A service targeting 99.9 percent availability has meaningfully more room for a slower, more careful failover than one targeting 99.99, where the allowed downtime shrinks to roughly 52.6 minutes a year¹.

Building for a higher target than your business actually needs is a real cost, in engineering time and operational complexity, that's worth questioning explicitly rather than defaulting to the most resilient possible design regardless of what the situation calls for.