How to Know If Your Agent Is Actually Working

Pull twenty to fifty real examples from actual usage, or from your own best guess at common cases if you haven't launched yet, and write down what a correct outcome looks like for each one. Resist the urge to write hundreds of cases before you've run any of them; a small set you actually review by hand teaches you more than a huge set nobody has time to look at.

Grade whether the agent called the right tool, whether its final action was correct, and whether it asked for clarification when it should have, rather than comparing its response text word for word against a reference answer. Two different but equally correct phrasings shouldn't fail a test; a wrong tool call should, even if the response text sounds confident.

For example, a refund agent's test case might specify the correct tool as the order lookup, the correct final action as declining the refund because the order is outside the return window, and clarification as not needed. Each part can be checked by code, and none depends on how the reply is phrased. When a case fails, the record of which part failed points straight at the prompt, the tool description or the model, which makes each failure faster to diagnose than a single overall score would.

Re-run your evaluation set whenever you change a prompt, a tool definition, or the underlying model, the same discipline you'd apply to a regression test suite for ordinary code. This is the single most effective way to catch a change that quietly makes the agent worse at something it used to handle well, a class of bug that's otherwise invisible until a customer hits it.

For tasks where correctness isn't a simple yes or no, tone, helpfulness, whether a summary captured the important points, use a second model call to grade the output against a rubric you write. This scales further than manual review, but treat its scores as a signal to investigate, not as ground truth on their own; spot-check the grader's judgments against your own periodically.

Use a different, ideally stronger, model for grading than the one being evaluated where you can, since a model judging its own outputs against a rubric it would also use to generate them tends to be more lenient than a genuinely independent check would be.

Build an evaluation framework in this order:

1. Collect twenty to fifty real examples, or your best guess at common cases, and write down what a correct outcome looks like for each.
2. Grade whether the agent called the right tool, took the right final action and asked for clarification when it should have, not whether the wording matches.
3. Re-run the set whenever a prompt, a tool definition or the underlying model changes.
4. Add a model-graded layer for subjective tasks, using a written rubric, and spot-check the grader against your own judgment.
5. Turn each real regression into a permanent case so the set grows one failure at a time.

Say a team updates their agent's system prompt to make its responses more concise, based on feedback that answers were running too long. The change reads well in a quick manual check, three or four example conversations all look tighter and clearer. Running the full evaluation set afterward tells a different story: on eleven of the forty cases, the more concise version now skips a required disclosure the longer version used to include, because the instruction to be brief and the requirement to state the disclosure were competing for space in the model's response.

Without the evaluation set, this regression would most likely have surfaced through a customer complaint or, worse, a compliance issue, weeks after the prompt change shipped. With it, the team caught the tradeoff the same day and rewrote the prompt to make the disclosure non-negotiable rather than something concision could quietly crowd out.

That eleven-out-of-forty case also became a new, permanent entry in the evaluation set itself, so the same tradeoff between brevity and the required disclosure gets checked automatically on every future prompt change, rather than depending on someone remembering this specific incident the next time concision comes up as a goal. This is how an evaluation set is supposed to grow over time, one real regression at a time, rather than being designed exhaustively up front by guessing at every case that might eventually matter, and it's a large part of why the set gets more valuable the longer a team maintains it.