Building a CI/CD Pipeline for Agent and Tool Code

Store prompts and MCP tool definitions in the same repository as the code that uses them, under normal version control, rather than in a separate content management tool that bypasses code review. This gives you a diff on every change, a blame history when something regresses, and the ability to roll back a bad prompt exactly the way you'd roll back a bad deploy.

A practical detail is to keep the prompt text in its own file, not inline in application code, so a wording change produces a small readable diff, and to require the same reviewer sign-off you would ask for on a code change. Tag each release with the prompt and tool schema versions it shipped with, so a trace from production can be matched to the exact text that produced it. When a regression appears, that pairing turns a vague suspicion about the agent into a specific commit to inspect or revert.

Before a prompt, tool, or model change merges, run it against your evaluation set automatically and block the merge if scores drop below a set threshold. This is the agent-specific equivalent of a unit test suite, and it catches the class of regression that's otherwise invisible until a customer hits it, an agent that quietly got worse at a task it used to handle well.

Roll a new prompt or tool version out to a small percentage of traffic first, watch tool error rates and fallback rates, then expand. Deployment frequency is a good proxy for how well this discipline is actually working: teams with strong practices ship on demand in small increments, while the slowest go as long as 180 days between releases¹. The gap between those two isn't really about tooling, it's about whether small, frequent, closely watched changes are the default or the exception.

Because a bad prompt change degrades quality rather than crashing, it can take longer to notice than a normal outage, which makes a fast rollback path even more important. Keep the previous version of every prompt and tool definition one command away from being restored, and practice the rollback occasionally so it's not the first time anyone's run it during an actual incident.

A rollback drill costs an hour every quarter or so and consistently turns up something worth fixing: a script that assumed a manual step nobody remembered, a dependency that wasn't actually pinned to the old version along with the prompt. Finding that during a drill is a minor annoyance; finding it during a real incident is a much longer night.

A pipeline for prompt and tool changes runs in this sequence:

1. Keep prompts and tool schemas in the same repository as the code, under normal version control and code review.
2. Run the evaluation suite automatically and block the merge if scores fall below your threshold.
3. Roll the change out to a small percentage of traffic and watch tool error rates and fallback rates against the control group.
4. Expand in stages only if those signals stay normal, and roll back with a single command if they do not.
5. Practice the rollback occasionally so it is not the first time anyone has run it during a real incident.

Say a new prompt version merges after passing the evaluation suite, and rolls out to five percent of traffic as the pipeline requires. Within the first hour, tool error rates for that slice tick up noticeably compared with the control group still on the old prompt, even though the evaluation suite hadn't caught anything wrong. The gradual rollout, not the evaluation suite, is what caught this one: the new prompt had changed how the agent formatted a date field passed into a downstream tool, a case the evaluation set's examples happened not to cover.

Because only five percent of traffic was affected and the rollback took a single command, the fix cost a few dozen confusing tool calls instead of a day's worth across the whole user base. The date-formatting case was added to the evaluation set immediately afterward, closing the specific gap for next time. None of this required anything exotic, just the discipline of watching the right metrics during a small, deliberate rollout instead of trusting a passing test suite to be the final word. A passing evaluation suite tells you the cases you thought to test still work; a gradual rollout with real monitoring tells you about the cases you didn't think to test, which is a different and equally necessary kind of coverage.