Writing an Incident Runbook for When Agents Misbehave

Write down the specific signals that should trigger an agent incident response, separate from your standard uptime and error-rate alerts: a spike in fallback responses, a spike in a specific tool's error rate, or a report that the agent took a destructive action incorrectly. Without this list, an agent behaving badly can go unescalated because nothing about it trips a traditional outage alert.

The first response to a misbehaving agent is usually not a code fix, it's disabling the specific capability that's causing harm, a single tool, or the agent entirely, while the team investigates. Build this kill switch before you need it, and make sure whoever is on call actually knows where it is and has practiced using it.

A kill switch only helps if the person holding the pager can find it and trust it. For example, run a short tabletop exercise once in a while: describe a misbehaving tool, then ask the on-call engineer to show, without help, how they would disable just that tool and how they would confirm it stopped. Any hesitation shows you where the documentation is missing. A common mistake is building a switch that turns off the entire agent only, which makes people reluctant to use it. Prefer switches at the level of a single tool or capability so the response stays proportionate to the problem.

Use these first-response steps when an agent misbehaves:

1. Match the report against your written list of agent incident signals and declare an incident, rather than waiting for a standard uptime alert to fire.
2. Disable the specific tool or capability causing harm through the kill switch, or the whole agent if the scope is unclear.
3. Pull the full decision traces for the affected sessions to see which tools ran, with which arguments and results.
4. Find the root cause from those traces instead of trying to recreate the conversation from scratch.
5. Add the failing case to the evaluation suite before you turn the capability back on.

Once the immediate behavior is stopped, pull the full decision trace for the affected sessions: which tools were called, what arguments were passed, what results came back. This is usually the fastest path to root cause, faster than trying to reproduce the exact conversation from scratch, and it's only possible if you were already logging full traces before the incident happened.

Before turning the affected capability back on, add the incident's specific case to your evaluation suite so it's tested automatically on every future change, not just fixed once and hoped never to recur. High-availability practice budgets a fixed amount of downtime per year depending on the target, for example roughly 8.76 hours annually at a 99.9 percent target¹; treat an agent incident's resolution time against a similar discipline, moving fast, but not so fast that the fix goes out untested.

Write a short summary of what happened and why, in plain language, even for a minor incident. The habit of writing these down consistently, not just for the dramatic ones, is what makes a pattern across several small incidents visible before it becomes a large one.

Say an agent starts approving a small category of refund requests it shouldn't, not because of any single obvious bug, but because a recent tool change subtly altered how it interprets a discount eligibility field. Nothing about this trips a standard uptime alert, the system is fast and responsive throughout. It surfaces because a finance team member happens to notice an unusual pattern in a weekly report, three days after the underlying change shipped.

Having a runbook ready meant the response was immediate once someone flagged it: disable the refund-approval capability through the existing kill switch within minutes, pull the decision traces for the affected sessions to confirm the scope, and identify the root cause in the tool change rather than guessing at several possible explanations. Without a runbook already in place, each of those steps would have needed to be improvised in the middle of an incident that was already three days old by the time anyone noticed, with no clear owner and no agreed first move while the affected capability kept running. The finance team member who first spotted it also became part of the postmortem, since a downstream team noticing an anomaly before engineering does is itself a signal worth building a channel for. That specific gap, no automated alert existed for this pattern at all, became its own follow-up item, separate from the fix to the discount eligibility logic itself, and it's now one of the standard signals the team watches after any change touching pricing or eligibility rules.