How to Build a Test Set That Actually Catches Bad Model Updates

• Real production examples, sampled regularly, not just the handful of cases someone wrote by hand when the feature launched.
• Known hard cases: past incidents, edge cases that broke a previous model version, and inputs your support team flags as tricky.
• A mix of formats, if your endpoint handles more than one kind of request, weighted roughly by how common each one actually is in production.
• Expected outputs or scoring criteria for each case, written down clearly enough that a different person could grade it the same way.

A test set that's only the second category, hard cases, catches regressions but tells you nothing about whether typical requests still work well.

Automated scoring, exact match, a rules-based check, a smaller model grading the larger one's output, is fast enough to run on every deploy and catches obvious regressions cheaply. It's also blind to a subtler failure: a response that's technically correct but worse in tone, or right on the surface but wrong about something a rule can't check.

Human review catches what automated scoring misses, at a cost in speed that makes it impractical for every single deploy. A workable split: automated scoring gates every deploy, and a sample goes to human review on a regular cadence, with any new failure pattern it finds turned into an automated check for next time.

Say a new model version passes every automated check, exact-match accuracy on the test set holds steady, but a sample of human review flags a pattern: the model has started giving noticeably shorter answers to a specific category of question, technically correct, but less useful than before.

Because that pattern wasn't in the automated test set, it would have shipped clean. Adding a length or completeness check for that category, once the pattern is identified, turns a one-time catch into a permanent guardrail for every future deploy.

Set a fixed cadence, monthly is reasonable for most teams, to pull a fresh sample of real traffic into the test set and retire cases that no longer reflect current usage. Tie it to your regular model review rather than treating it as a separate project that competes for attention.

The alternative, letting the test set calcify, is worse than having a smaller one that actually gets refreshed. A stale comprehensive test set gives false confidence; a smaller current one at least tells you the truth about recent traffic.

If more than one person does human review, agree on scoring criteria in writing before you start, and periodically have two people grade the same sample independently to check they land on similar scores. Without that check, your evaluation results reflect whoever happened to review that batch as much as they reflect the model.

A short rubric, even a rough one, beats no rubric; it turns a subjective judgment call into something a new reviewer can pick up consistently.

• Testing only the happy path and skipping the edge cases that caused your last real incident.
• Letting the same person who built the model also be the only one grading its outputs, with no second reviewer.
• Treating a passing test set as proof of quality rather than as one signal among several, including real user feedback after the fact.

A test set is a floor, not a guarantee; treat a pass as permission to ship carefully, not as a finished verdict.