Where AI Inference Costs Actually Go, and What to Cut First

• Context length: every extra token you send costs money on both input and output; trimming unnecessary context in your prompt template often beats a smaller model.
• Batching and concurrency: running requests through continuous batching instead of one at a time changes GPU utilization more than almost anything else you can do.
• Autoscaling floor: a minimum replica count set for worst-case traffic burns money the rest of the day; set it for typical load and let it scale up.
• Model size and quantization: the lever everyone reaches for first, and the one with the most obvious quality trade-off, which is why it should come after the other three, not before.

Say your team spends $40,000 a month on inference for a single customer-facing feature. A quick audit finds the prompt template sends 2,000 tokens of boilerplate instructions on every call, batching is effectively disabled because requests are proxied one at a time, and the GPU pool runs at a fixed size around the clock.

Cutting the boilerplate to 400 tokens, enabling batching so the same GPUs handle more concurrent requests, and letting the pool scale down overnight can each independently shave a meaningful slice off that bill, before anyone touches the model itself. Stack all three and the total savings usually dwarfs what switching to a cheaper model alone would have delivered.

Once inference isn't a single team's line item, allocation gets political fast. The cleanest approach is a shared pool with per-team quotas and chargeback based on actual tokens processed, not a fixed monthly allotment that goes unused by one team and blocks another.

Set the quota as a ceiling with an approval path to raise it temporarily, rather than a hard wall that forces a team to file a ticket every time it runs a larger batch job. Review actual usage against quota monthly; a team consistently under its quota is a sign the quota was set from a guess, not from data.

It sounds backward, but a larger GPU can lower total cost per request when your workload is memory-bound rather than compute-bound: fitting a bigger batch, or a longer context window, onto fewer, pricier GPUs sometimes beats spreading the same load across more, cheaper ones once you count the overhead of running more machines.

The way to check is cost per thousand tokens processed at your actual batch size, not the sticker price per GPU-hour. A cheaper GPU that forces smaller batches can lose on that measure even though it wins on the hourly rate.

• Switching to a smaller model without re-running your evaluation gate, which can quietly increase the number of retries or fallback calls that eat the savings.
• Caching responses aggressively without a clear invalidation rule, which serves stale answers and creates a different cost: support tickets.
• Rightsizing GPUs based on average load instead of your actual peak, which saves money until the first traffic spike causes a queue backup that costs you customers.

Every one of these trades a visible line item for a less visible one.

Cloud dashboards default to showing GPU spend per hour, which is easy to read and easy to misuse. Two GPU types can have very different hourly rates and still cost about the same per useful request once you factor in how many requests each one can actually serve at your latency target.

Track cost per thousand tokens processed, or cost per completed request, as your primary metric, and treat the hourly rate as a secondary number you check when comparing options. A team that manages cost per hour instead of cost per request keeps making decisions that look good on a spend report and bad in practice.