Why Your RAG SLA Alerts Stop Firing Right When You Need Them

Say your SLA is a 3-second response time at the 95th percentile. Under normal load, the embedding call takes 200 milliseconds, the vector search takes 100 milliseconds, reranking takes 300 milliseconds, and generation takes 1.5 seconds, comfortably inside budget. Now the embedding provider has a partial outage and its latency climbs to 4 seconds for a subset of requests. Your end-to-end p95 crosses the SLA threshold, but if your alerting only watches the end-to-end number, it fires a generic slow response alert that tells the on-call engineer nothing about where to look first.

Break your SLA into a budget per hop, embedding, vector search, reranking, generation, and alert on each one independently, in addition to the end-to-end number. When a hop's latency crosses its own budget, the alert names the hop, which cuts the time to diagnose from a broad investigation to a targeted one. This is more instrumentation work up front than one end-to-end check, but it's the difference between an alert that tells you what's wrong and one that tells you something is wrong.

A 95th percentile SLA, by definition, ignores roughly one in twenty requests, the worst ones, and that slice is often concentrated: the same users, the same query types, or the same time window, hitting a real problem every time while the aggregate metric looks fine. If your corpus has a subset of documents that are unusually large or your query mix includes a class of question that needs more reranking passes, that subset can sit permanently outside your SLA while the dashboard stays green. Track SLA compliance by query segment, not only in aggregate, if you have segments that behave differently.

Automated SLA enforcement should catch breaches faster than a human watching a dashboard, and it should not automatically take destructive action, like failing traffic over to a degraded fallback, without a person confirming the failover is actually the right call for that specific breach. A retrieval system has enough failure modes, a slow embedding provider, a corrupted index, a cold cache, that an automated response tuned for one can make a different one worse. Alert fast, act with a person in the loop.

• Measuring only the end-to-end latency, so an alert never names which hop is actually slow
• Setting the SLA threshold once at launch and never revisiting it as the query mix changes
• Ignoring the worst slice of requests because the 95th percentile metric already excludes them by definition
• Wiring automated alerts to automated remediation without a person confirming the action fits the specific failure
• No segment-level view, so a consistently broken subset of queries hides inside a healthy aggregate

The SLA number you set at launch was a guess based on whatever traffic and corpus you had then. Revisit it as the corpus grows, the query mix shifts, or you add a reranking step that wasn't there before, and update both the threshold and the per-hop budgets that support it. An SLA nobody has revisited in a year is either too loose to mean anything or too tight to be realistic, and neither is useful during an actual incident.

A useful rule: any time you change the shape of the pipeline, treat the SLA and its per-hop budgets as open for review. Adding a reranking step, for instance, adds latency that has to come from somewhere, so either the other hops give up some budget or the end-to-end threshold moves. Decide which explicitly, write the new per-hop numbers down, and update the alerts in the same change. Otherwise the old budgets quietly stop describing the system you actually run, and the first alert to misfire, or fail to fire, will happen during an incident.