When Multi-Region Routing Sends Traffic to the Wrong Place

Most routing layers check the thing that's easiest to check, not the thing that matters. A TCP handshake or a shallow health endpoint can return green while the database connection pool behind it is exhausted, or a downstream dependency in that region is timing out. The router keeps sending traffic to a region that's technically up and practically unusable.

Deep health checks fix this, but they carry their own risk: a check that calls three downstream services to verify itself, and then times out slowly, can make the health signal the bottleneck during an incident. Aim for a check that verifies the two or three dependencies that actually gate a successful request, with a strict timeout, and nothing beyond that. Watch your DNS caching too: a health check that flips in seconds does nothing if resolvers and clients are still holding the old answer for minutes.

Active-active architectures promise better utilization and faster failover, and they deliver both, right up until a network partition lets two regions each believe they own the same write. Whichever conflict-resolution rule you picked, last-write-wins, vector clocks, a designated tiebreaker region, gets exercised for real for the first time during an outage, which is the worst possible moment to discover it's wrong.

If your data layer can't genuinely support multi-writer semantics for a given table, don't pretend it can with routing tricks. Pin writes for that data to a single region and make failover to a backup writer an explicit, monitored, and deliberately slower operation than read failover. A brief pause in write availability during a real regional failure beats silently corrupted data that nobody notices until reconciliation.

Sticky sessions and in-region caches are the part of multi-region design nobody wants to think about, because reasoning about routing tables is more interesting than reasoning about a shopping cart that lived only in a regional cache. When a cutover happens, those sessions either get dropped, silently reconstructed with stale data, or, worst case, partially replayed against the new region's state.

Decide, deliberately, what a user experiences when their session's home region disappears mid-request: a forced re-login, a rebuilt cart pulled from durable storage, or a short, visible retry. Whatever you pick, make it consistent and make it something support can explain in one sentence, instead of something engineering discovers three tickets into an incident.

If you've committed to 99.9 percent uptime, you're working with well under a full day of downtime for the entire year, about 8.76 hours total¹, and a botched regional cutover can burn through that budget in minutes. The safest way to learn how your routing actually behaves under failure is to fail a region over on purpose, on a schedule, with someone watching.

Start by shifting a single-digit percentage of read traffic to the secondary region during low-traffic hours and confirm the health signal, the dashboards, and the alerting all agree with reality. Widen the traffic share, then include writes, then finally schedule a drill that takes a region fully out of rotation for a defined window. Each step should leave a short written record of what broke, because the value of a drill is the list of assumptions it corrects, not the successful run.

A surprising number of multi-region setups keep their routing configuration, feature flags, or DNS management console dependent on the very region that's most likely to fail. When that region goes down, the team can see the problem but can't route around it, because the control plane for fixing it lives inside the blast radius.

Host your routing control plane, your DNS provider access, and your incident runbooks somewhere that survives the failure you're planning for. That usually means a third-party DNS or traffic-management service, credentials that don't depend on regional single sign-on being up, and a runbook stored somewhere reachable when your primary tooling isn't.

Before you trust a failover, check each of these:

• Health checks test the dependencies that matter, such as the database connection pool, instead of a shallow handshake that stays green.
• Only one region can own a given write during a network partition, so two regions never both act as primary.
• Sessions and in-region caches have a plan for what happens to requests that were mid-flight during the cutover.
• DNS caching is accounted for, since real users and third-party resolvers keep a cached answer long after your dashboards update.
• Routing configuration, feature flags and the DNS console do not depend on the region most likely to fail.