Circuit Breakers and Bulkheads, Explained With Checkout

After a threshold of failures or timeouts against a dependency, the breaker opens and starts failing fast on calls to it for a cooldown period, instead of letting every caller wait out the full timeout on a dependency that's already known to be failing. This protects your own service's limited resources, threads and connections, from being consumed waiting on something that isn't going to answer anyway.

After the cooldown period, a well-behaved breaker lets a small number of trial requests through to test whether the dependency has actually recovered, rather than either staying open forever or slamming straight back to fully closed. A common bug is a breaker that reopens the instant that very first trial request fails, which under any residual flakiness in the dependency means it effectively never recovers on its own.

A bulkhead gives each downstream dependency its own bounded pool of threads or connections, so a stuck call to one dependency can exhaust its own pool without touching the pool serving unrelated requests. Without that isolation, threads blocked waiting on a slow payment provider are the exact same threads needed to serve a completely unrelated product page request, which is how one dependency's problem becomes everyone's problem.

A threshold that's too sensitive trips the breaker on ordinary latency variance, opening it for a dependency that was actually fine and unnecessarily degrading a feature that didn't need to degrade. A threshold that's too loose never trips before your own service is already in real trouble. Base it on the dependency's own observed failure rate and latency distribution from real traffic, not a number that sounded reasonable in a planning meeting.

A useful decision rule is to set each breaker from that dependency's own history. Look at how often it times out during a normal week and how slow its responses get at busy hours, then set the trip point clearly beyond that normal variation. For example, a tax calculation service that regularly slows down at month end should not share a threshold with a payment provider that is normally steady. A common mistake is copying one threshold across every dependency. Review each threshold after any incident where a breaker tripped late, or tripped for no good reason.

A checkout service calls three downstream dependencies: inventory, payments, and tax calculation. Each gets its own breaker and its own bulkhead. When payments starts timing out, its breaker opens and its bulkhead's threads stay exhausted only within that isolated pool. Inventory and tax calls, along with the parts of checkout that don't depend on payment confirmation, keep serving normally while payments degrades gracefully behind a clear fallback message instead of taking the entire checkout flow down with it.

• Applying a circuit breaker without a bulkhead, so failures get detected correctly but shared resources can still be starved by the failing dependency.
• Setting one global timeout for every dependency instead of one per dependency, so a fast dependency's timeout ends up either too aggressive or too lax for a genuinely slower one.
• Never deliberately testing the open and half-open states before you need them, so the first real test of the whole mechanism happens live, during an actual incident.

A raw error thrown all the way up to the end user is rarely the right fallback. For checkout specifically, a clear message that payment processing is temporarily unavailable, with an obvious way to retry shortly, treats the customer better than a generic failure and buys you the cooldown window without losing the sale outright.

Design the fallback deliberately per dependency instead of letting whatever exception happens to surface become the user-facing message by accident. A payments outage and an inventory-lookup outage warrant genuinely different fallback behavior, and that difference is worth designing on purpose rather than discovering live during the next incident.