What to Actually Alert On When You Serve Models in Production

• Output length distribution. A sudden shift toward much shorter or much longer responses often shows up before users complain.
• Refusal or fallback rate. How often the model declines, times out, or falls back to a default response, tracked as a trend, not just a raw count.
• Token-level cost per request. A spike here without a matching traffic spike usually means something changed in your prompt template or the model's behavior.
• GPU memory pressure and queue depth, which predict a latency problem before it shows up in your latency graphs.

None of these need a research team to build; they're mostly a few extra fields logged on every request.

An error budget gives you a number to alert against instead of a gut feeling. If your target is 99.9 percent availability, you're working with roughly 8.76 hours of allowed downtime for the whole year¹; at 99.95 percent, that budget shrinks to about 4.38 hours.

Once you know your annual budget, break it into a monthly or weekly allowance and alert when you're burning it faster than a steady pace would allow, not just when you hit zero. That catches a slow leak, a service that's degraded but not fully down, before it consumes the whole year's budget in a bad month.

Start with three tiers. Page immediately on infrastructure failures: the endpoint is down, GPU nodes are unreachable, or queue depth is climbing without bound. Alert during business hours on quality signals: refusal rate above its normal range, output length distribution shifting, or token cost per request rising without a traffic increase to explain it. Log everything else for weekly review: small drifts that aren't urgent but are worth a human look before they compound.

The mistake most teams make is putting all of this in the first tier, which trains everyone to ignore alerts, or leaving it all in the third tier, which means quality drift gets caught a week after users noticed.

When something goes wrong, you want to answer one question fast: was this the model, the prompt, or something upstream. That requires tagging every logged request with the model version, prompt template version, and any retrieved context or tool results involved, not just the final output.

Without that tagging, a support ticket about a bad answer turns into an afternoon of guessing. With it, you can filter for every request that used the same model and prompt version and see whether the issue is isolated or systemic within minutes.

• Raw GPU temperature, unless you're managing your own hardware; your cloud provider already handles this layer.
• Every individual prompt and response in a real-time dashboard; sample instead, and log the rest for offline review.
• Model confidence scores in isolation, when the model doesn't have a well-calibrated confidence signal to begin with; a raw score without calibration is noise dressed up as a metric.

Monitoring everything is the same mistake as monitoring nothing: both leave you unable to tell a real signal from background noise.

Traffic patterns and model behavior both drift over time, and a threshold that made sense at launch can become useless within a quarter. A refusal rate alert tuned for a slow initial rollout will stay silent through a real problem once traffic grows, because the raw counts look small against the new baseline even though the underlying rate is unhealthy.

Revisit your alert thresholds on a fixed schedule, not only after an incident forces the review. A short monthly check against the previous month's actual data catches drift in the thresholds themselves before it becomes a blind spot in production.