Setting Rate Limits and Spend Caps Without Breaking Real Users

• A request-rate cap, protecting infrastructure, set close to what your GPUs can actually handle before latency degrades for everyone.
• A spend cap, protecting budget, set per caller or per feature based on expected usage, independent of whether the infrastructure could technically handle more.

A caller can be well under the request-rate cap and still blow through a spend cap if their requests are unusually large, long prompts, high output token counts, so track both separately rather than assuming one implies the other.

Checking the limit at your API gateway, before a request reaches the model server, is cheaper and catches abuse earlier, but it needs to know enough about the request, expected token count, which model it's headed for, to make a meaningful decision.

Checking it at the model server catches everything the gateway might miss, but by then you've already paid for whatever work happened before the check. The practical answer is both: a coarse check at the gateway to reject obvious abuse fast, and a finer per-model, per-token check closer to where the actual cost is incurred.

Work through four numbers for each caller type: typical requests per day, typical tokens per request, your cost per thousand tokens, and the multiple above typical usage you're comfortable allowing before you want a human to look at it, often somewhere between three and five times normal.

Multiply the first three together for a typical daily cost, then apply your comfort multiple for the actual cap. Review actual usage against this cap monthly and adjust it; a cap set once from a guess and never revisited either blocks legitimate growth or fails to catch a real problem.

A hard rejection is the simplest response and the easiest to get wrong for a legitimate customer having an unusually busy day. Consider a soft response instead for the spend cap specifically: throttle rather than reject outright, and notify someone on your team so a human can decide whether to raise the cap.

Reserve hard rejection for the request-rate cap protecting infrastructure, where the risk of letting traffic through unchecked is a real outage rather than an unexpected bill you can review after the fact and, if needed, follow up on with the customer directly.

A useful decision rule is to ask what the worst outcome is if the request goes through. If the worst outcome is an outage for every other customer, reject it. If the worst outcome is a larger bill for one caller, throttle and notify. For example, a customer who starts a large batch import can be slowed to a lower rate with a message explaining why, while a runaway loop sending the same request over and over should be blocked at the gateway. Writing that rule down means the on-call engineer is not inventing policy in the middle of an incident.

A caller who hits an undocumented limit assumes something is broken, not that they've been capped. Publish your rate limits and, where practical, your spend-cap defaults, so integrators can design around them instead of discovering them through failed requests.

For spend caps specifically, consider a usage dashboard or a proactive notification well before the cap, at a comfortable majority of it, so a customer scaling up gets a heads-up instead of a surprise rejection at the worst possible moment for whatever they're building on top of you.

• One global limit applied to every caller regardless of their actual usage pattern or contract.
• No monitoring on how close callers run to their limit, so you find out about a problem only after they've already been blocked.
• Treating a spend cap breach the same as a security incident, when it's usually just a customer using the product more than expected.

The goal is protecting the business, not punishing growth. A customer who outgrows a cap because their own usage is genuinely growing is a sales conversation, not a security event.