Keeping Agent Workflows Running When a Region Goes Down

For every MCP server and every external API your agents call, write down three things: what the agent should do if that dependency times out, what it should do if it returns an error, and whether the task can complete at all without it. A tool that looks up a customer's order history might be skippable, with the agent noting it couldn't check, while a tool that processes a payment cannot be skipped without the task simply failing.

This worksheet turns a vague goal like "the agent should be resilient" into a specific, reviewable list of decisions.

Higher availability costs more in engineering time and infrastructure, so match the target to what the workflow actually needs. A target of 99.9 percent leaves roughly 8.76 hours of downtime a year to work with, while 99.99 percent leaves around 52.6 minutes¹. An internal agent that helps engineers triage tickets can usually live with the looser budget; an agent handling live customer conversations usually can't.

A model provider outage or a downstream MCP server outage doesn't have to take the whole agent down. Build the agent to recognize which of its tools are unavailable and either complete a reduced version of the task or hand off to a human with a clear explanation of what it couldn't check, rather than returning a raw error or, worse, guessing at an answer it can't actually support.

For example, an agent that summarizes support tickets can respond to a model provider outage by queuing the request and telling the user the summary will arrive shortly. An agent that approves refunds should stop and route to a person instead, because a delayed answer costs little while a guessed approval costs real money. Writing the degraded behavior next to each row of the worksheet, including the message the user sees, keeps the response the same no matter which conversation hits the outage first, and lets you review it before an incident instead of improvising during one.

Run a scheduled drill where you deliberately disable one MCP server or simulate a model provider timeout and watch what the agent actually does. Plans that look reasonable on paper often reveal a tool with no timeout at all, one that will hang the whole agent loop rather than fail fast, and you only find that by testing it.

Write down what you expected to happen before you run the drill, then compare it against what actually happened afterward. The gap between the two is usually where the real work is, and it's a much cheaper place to find that gap than during an actual outage with real customers waiting on a response.

Run a failure drill in this sequence:

1. Write down what you expect the agent to do before you run anything, so the result has something to be compared against.
2. Deliberately disable one MCP server or simulate a model provider timeout.
3. Watch what the agent actually does, including whether any tool hangs the whole loop instead of failing fast.
4. Compare the outcome with your written expectation, since the gap between the two is usually where the real work is.
5. Fix the gaps, then repeat the drill after adding any new MCP server that other tools now depend on.

Say your agent stack has one MCP tool that looks up shipping status from a third-party carrier API. For a customer-facing tracking assistant, the fallback might be to tell the user the agent couldn't reach the carrier right now and to try again shortly, since a wrong shipping estimate is worse than an honest "I don't know." For an internal support agent helping an employee investigate a complaint, the fallback might instead be to proceed with a note that shipping status couldn't be confirmed, since the employee can judge for themselves whether that gap matters for the specific case. For a fully automated workflow that issues refunds based partly on shipping delay, the fallback has to be to stop and route to a human, because acting without that data at all is the one option that's actually unsafe.

Same dependency, same failure, three different correct answers, because the right fallback depends entirely on what happens downstream of the agent's decision, not on the tool itself. Writing all three down explicitly, rather than leaving the agent to improvise a response to the same outage in each context, is what makes the difference between a graceful degradation and an inconsistent one that varies by which conversation happened to hit it first.