Setting Spend Caps on a RAG Pipeline Without Breaking It

A request-based cap is simple for users to understand but a poor proxy for cost, since one request with a long document and a large top-k can cost far more than ten simple ones. A token-based cap tracks cost accurately but is harder for a user to reason about in advance. Many teams end up with both: a token-based cap for actual cost control, translated into a simpler request-based indicator shown to users so the limit doesn't feel arbitrary.

Whichever unit you choose, measure it consistently across every stage of the pipeline. A cap that counts tokens at the generation step but ignores tokens spent on embedding and reranking will systematically undercount real cost, and undercounting compounds quietly until the actual bill stops matching what the cap implied it should be.

A single global cap means one tenant running an unusually heavy workload can consume the entire budget and degrade the experience for everyone else sharing it. Per-tenant caps, sized to each tenant's actual plan or usage tier, with a shared overflow pool for genuine bursts, keep one noisy tenant from starving the rest while still letting you use spare capacity efficiently rather than leaving it idle.

For example, suppose two tenants share one budget and one of them starts a bulk workflow that embeds long documents with a large top-k. Under a single global cap, the second tenant's ordinary queries start failing even though nothing about their usage changed. With a per-tenant cap and a shared overflow pool, the heavy tenant reaches its own ceiling and draws on spare capacity, while the second tenant is unaffected. The team can also see exactly which tenant drove the spend, which makes a plan upgrade conversation concrete instead of speculative.

Queueing a request past the limit adds latency but preserves the full experience once it processes, which fits a background or batch workflow where a short delay is acceptable. Degrading gracefully, skipping the reranking pass, retrieving fewer chunks, still returns something useful and fits an interactive feature where users expect a response now. A hard rejection is the simplest to implement and the worst experience, and it's worth reserving for genuine abuse rather than routine peak usage.

The three responses at the limit suit different workloads:

• Queue the request when a short delay is acceptable, such as background or batch work, so it still gets the full experience once it runs.
• Degrade gracefully by skipping the reranking pass or retrieving fewer chunks when users expect an answer right now.
• Reject outright only for genuine abuse, since it's the simplest to build and the worst experience for routine peak usage.

If ingestion and query-time embedding share the same provider quota, a large bulk upload can silently starve query traffic that has nothing to do with it, showing up as a mysterious latency spike with no obvious cause in the query path itself. Give ingestion its own quota, or explicitly prioritize query traffic over ingestion when both compete for the same upstream capacity, so a backfill job never degrades the live user experience.

A usage indicator, or a warning once a tenant crosses most of its allotment, turns a hard cutoff from a surprise into an expected event. This is a small addition on top of the underlying rate limiting logic, but it's the difference between a limit that feels like a broken feature and one that feels like a plan boundary the user already understood.

A cap sized only around embedding and search cost will miss the fact that reranking and generation usually cost more per request than the retrieval step itself. If your rate limiting logic only tracks the initial search call, a tenant can still drive up spend through expensive reranking or long generation outputs without ever tripping the limit. Track cost across the whole request path, not just the stage that happens to be easiest to instrument first.

A cap sized around a tenant's usage at signup tends to become stale within a few months, either too tight for a tenant that's grown into heavier legitimate use or too loose for one whose usage pattern shifted toward more expensive query types. Review caps on a regular schedule against actual usage data, rather than only adjusting them reactively when a tenant complains or a cost spike forces the question.