Catching Broken Tool-Calling Schemas Before They Reach Production

• Schema conformance on every field, not just the ones a quick manual check happens to look at, including optional fields the model sometimes omits.
• Tool-calling format, if your model can invoke tools or functions, checked against the exact schema your application code expects to parse.
• Type stability, catching a model that returns a number as a string on some responses and a number on others, which breaks strict parsers silently.
• Behavior under malformed or edge-case input, not just well-formed requests, since that's where schema drift shows up first.

Run these on a schedule against the live model, not only at integration time, since model behavior can shift between deploys on the provider's side too.

If you call a third-party model API, a provider-side update can change formatting behavior without any warning reaching you, because from their side it's an improvement, not a breaking change. Your application code, which was written against the old formatting quirks, can start failing to parse responses it used to handle fine.

This is the strongest argument for running contract tests continuously rather than once: a test suite that only runs when you touch your own code will never catch a regression that originates entirely on the provider's side.

Tool-calling output deserves its own test category, separate from general response schema checks, because it tends to be more structurally rigid and more consequential when it breaks: a malformed tool call can trigger the wrong action entirely, not just display incorrectly.

Build a fixed set of prompts designed to trigger each tool your application supports, and validate the resulting call structure against your parser's actual expectations, not just against the provider's published schema. Providers occasionally return technically-valid-but-unexpected structures that a strict published schema wouldn't flag as wrong.

Say a model provider updates their model and it starts returning dates as full sentences instead of a fixed format on a small percentage of requests, technically still valid text, but not what your date parser expects. Without a contract test specifically checking output format on that field, this ships silently and shows up as sporadic parsing errors that look random until someone traces them back to date formatting.

A test that runs the same prompt regularly and checks the date field's format specifically catches this within a day instead of within a support escalation weeks later.

A common mistake is writing a contract test that checks a response has the right fields and stops there. The fix is to assert on the values your code actually depends on. If a downstream function expects a date in one format, an enum from a fixed set, or a number rather than a string, test exactly that. Then feed the output through the same parser your application uses, so the test fails the way production would fail. A contract test that only inspects shape will keep passing while your parser quietly breaks.

Keep your fixed test prompts and expected schema versioned in the same repository as your application code, not in a separate document that drifts out of sync. When you update the contract deliberately, update the fixtures in the same pull request, so the two never diverge without someone noticing.

A test suite that still checks against a contract you retired months ago produces false confidence: it passes reliably while checking something nobody actually relies on anymore, and nobody notices until a real regression slips through the gap it left behind.

• Testing only well-formed, happy-path requests, missing the edge cases where schema drift actually shows up first.
• Running contract tests only at deploy time for your own code, missing regressions that originate on a third-party model provider's side.
• Validating against a provider's published schema instead of your own parser's actual expectations, which can diverge in subtle ways.