Working Out Your Pipeline's Actual Downtime Budget

Availability targets sound abstract until you convert them into a downtime budget: at 99 percent that is about 3.65 days of allowed downtime a year, at 99.9 percent about 8.76 hours, at 99.95 percent about 4.38 hours, at 99.99 percent about 52.6 minutes, and at 99.999 percent about 5.26 minutes¹.

Each step up costs meaningfully more engineering effort and infrastructure than the last, so pick the target your pipeline's downstream dependents actually need, not the most impressive-sounding number. A pipeline feeding an internal analytics dashboard doesn't need the same budget as one feeding a live payment authorization flow.

Running brokers across multiple availability zones protects against a single zone failure, which is the most common infrastructure failure mode, but it doesn't protect against a bad deploy, a schema change that breaks every consumer at once, or a regional outage. Multi-AZ is table stakes; it's not a complete failover strategy on its own.

If your downtime budget is tight enough to require protection against a regional failure, that means active replication to a second region, which is a substantially bigger commitment in both cost and operational complexity than multi-AZ alone. Most pipelines don't actually need this; check your real budget from the last step before building it.

Recovery time objective (how long you can be down) and recovery point objective (how much data you can afford to lose) are different numbers, and a failover plan needs both. A stream processor that fails over instantly but loses the last few seconds of unprocessed events has a great RTO and a nonzero RPO; whether that's acceptable depends entirely on what the data is.

Say your pipeline feeds a real-time inventory count: losing a few seconds of updates during a failover is recoverable on the next sync. If it feeds a payment ledger, that same gap is not acceptable, and your failover design needs synchronous replication or an equivalent guarantee, not just a fast restart.

A failover plan that's never been exercised tends to fail on details nobody thought to write down: a hardcoded broker address, a consumer group that doesn't rejoin cleanly, a downstream system that can't handle a brief burst of reprocessed events. Run a real failover test on a non-production environment, on a schedule, not just after an incident forces you to.

Measure your actual RTO during that drill against the target you set, not the one you assumed. The gap between the two is usually where the next quarter's failover work should go.

The moment a failover happens, some events are mid-processing: read from the topic but not yet written to their destination. Decide in advance whether your consumers are built to safely reprocess an event they already handled (an idempotent write, keyed on an event ID so a repeat doesn't double-count) or whether a failover risks duplicate side effects downstream.

Idempotency is usually the cheaper fix than trying to guarantee exactly-once delivery through the failover itself, and it pays off beyond disaster recovery too, since the same protection covers ordinary consumer restarts and retries. If a downstream write can't be made idempotent (a webhook that triggers an external charge, say), that's the one path that needs its own dedupe logic before you can trust failover for it.

The technical steps matter less than most teams assume compared with a clear answer to a simpler question: who has the authority to declare a failover and start the process, at any hour, without waiting for a meeting. Ambiguity here costs more real downtime than any missing runbook step.

Name the role, not a specific person, so the answer still holds when that person is asleep or on vacation, and make sure whoever is on call actually knows they hold that authority before the night they need to use it.

To turn an availability target into a failover plan, work through these steps:

1. Pick the availability target your downstream dependents actually need and convert it into a yearly downtime budget.
2. Run brokers across multiple availability zones as the baseline, then judge whether your budget justifies more.
3. Set both a recovery time objective and a recovery point objective for the data the pipeline carries.
4. Make consumers idempotent, keyed on an event ID, so events in flight during the switch can be safely reprocessed.
5. Rehearse failover in a non-production environment on a schedule, not only after an incident.
6. Name the role that can declare a failover at any hour without waiting for a meeting.