A Rollout Checklist for Swapping Models in Production

A broken code deploy usually throws errors, fails a health check, and gets rolled back automatically within minutes. A broken model deploy often returns valid-looking responses that are just worse: less accurate, more prone to a specific failure mode, or subtly off for one segment of traffic.

That means your standard health check, does the endpoint return a success code, isn't enough. You need an evaluation gate that runs before traffic increases, checking the new model against a fixed test set and comparing its outputs to the version it's replacing. A model that passes every infrastructure check and still fails your evaluation gate should block the rollout the same way a failed test suite would.

Three rollout patterns work well for models, for different reasons:

• Shadow traffic: send a copy of production requests to the new model without serving its responses, so you can compare outputs before anyone is affected.
• Canary: serve the new model to a small, fixed slice of real traffic and watch both infrastructure metrics and output quality before expanding it.
• Blue-green: run both versions fully in parallel and switch traffic at the load balancer, which gives you an instant rollback but doubles your GPU cost during the cutover.

Most teams should default to shadow traffic first, then canary. Blue-green is worth the extra cost mainly for a model change large enough that you don't trust a canary sample to catch the risk.

Beyond the evaluation gate, confirm four things before a model swap goes past canary: the new model's latency profile under real load, not just a benchmark script; that your prompt templates and any tool-calling schema still match what the new model expects; that logging and tracing tag requests with the model version, so a regression is traceable after the fact; and that you have a fast rollback path that doesn't require a full redeploy, usually a router flag or a weighted traffic split you can flip in seconds.

Skipping the schema check is the one that bites teams most often: a new model version can change how it formats tool calls or structured output even when the prompt hasn't changed.

Teams that deploy often tend to deploy safer, not despite the frequency but because of it: smaller changes are easier to evaluate and easier to roll back. DORA's research clusters engineering organizations by deploy frequency, from top-performing teams deploying on demand down to the slowest cluster deploying between once a month and twice a year¹.

Model deploys are a good place to apply the same logic. A smaller, more frequent model update is easier to gate and easier to clear when something goes wrong than a rare, large one that changes several things about the model's behavior at once.

• Rolling back the model but forgetting to roll back the prompt template that was tuned for its quirks, which breaks the old model too.
• Running the evaluation gate against a stale test set that doesn't reflect current traffic patterns.
• Treating a fine-tune of the same base model as low-risk by default; a fine-tune can shift behavior more than a version bump.
• Skipping shadow traffic for a small change, which is exactly the kind of change most likely to slip through a canary sample unnoticed.

The common thread: every one of these skips a check that felt optional under deadline pressure.