Failover and High Availability: The Questions Worth Asking First

Fewer than most teams assume. Going from 99 percent to 99.9 percent availability cuts allowed downtime from 3.65 days a year to about 8.76 hours, and going from 99.9 to 99.99 cuts it again to roughly 52.6 minutes¹. Each step up costs meaningfully more in engineering complexity, and for most products that first jump matters far more to customers than any of the jumps after it.

Pick the target based on what your contracts and your customers' patience actually require, not what sounds impressive in a pitch deck. A target you can actually hit and explain to a customer beats a more ambitious one that quietly slips every quarter.

Active-active, where multiple regions or instances all handle live traffic, gives you the fastest failover because there's nothing to promote, but it demands that your data layer handle writes from multiple locations without conflicts, which is genuinely hard and easy to get subtly wrong.

Active-passive, where a standby only takes over on failure, is simpler to reason about and cheaper to run, at the cost of a failover delay while the standby spins up and takes over. For most small and mid-sized teams, active-passive with a well-tested failover path beats an active-active setup nobody fully understands, since an architecture your team can actually operate under pressure is worth more than one that's theoretically faster but poorly understood.

Almost always, it's the parts nobody tested. DNS records with a TTL too long to fail over quickly. A standby database that's been silently falling behind on replication for weeks. Application code with a hardcoded reference to the primary region's endpoint that nobody remembered was there.

The failover mechanism itself usually works in a demo. What breaks it in production is a dependency nobody accounted for, discovered for the first time during the actual outage.

Start small and scheduled. Fail over a low-traffic service during a planned window, with the team watching, before ever trying it on anything customer-critical. Treat the drill like a real incident: time how long it actually takes, and write down every step that required a human to remember something instead of following a runbook.

A failover plan that's never been exercised is a hypothesis, not a plan. Run the drill quarterly at minimum, and after any significant architecture change to the services it covers.

A first failover drill can follow these steps:

1. Pick a low-traffic service for the first drill, not anything customer-critical.
2. Schedule the failover for a planned window and have the team watching while it runs.
3. Run it like a real incident, with the same roles and communication you would use in an actual outage.
4. Note what did not work, such as slow DNS changes or a standby that fell behind on replication, and fix it before the next drill.
5. Move on to more critical services only after the small drills go cleanly.

For internal tools, batch jobs that can retry later, or anything where a short outage costs inconvenience rather than revenue, full multi-region redundancy is usually overbuilt. Save the investment for the paths that are actually customer-facing and time-sensitive, and accept a simpler recovery process, restore from backup, restart from the last checkpoint, for everything else.

Matching the redundancy investment to what each service actually needs, rather than applying one standard everywhere, is usually the difference between a high-availability strategy that's affordable and one that isn't.

A standby that looks ready can still lose data on failover if it was behind the primary when the primary went down. Any write that hadn't replicated yet is gone, or worse, conflicts with what the standby thinks happened next once it takes over.

Monitor replication lag as its own signal, separate from whether the standby is technically online. A standby that's up but hours behind gives you a false sense of protection, and finding that out during a real failover is a much worse time to learn it than during a routine check.