The Real Cost Drivers in a RAG Pipeline

It's easy to budget for query-time embedding costs and forget that ingestion embeds every chunk of every document, often repeatedly if your pipeline re-embeds on every update rather than only on change. If you're re-indexing a full corpus nightly instead of incrementally updating changed documents, you're paying to re-embed unchanged content every single day. Check your ingestion logs for how much of your embedding spend is redundant re-embedding versus genuinely new content.

A content hash comparison before re-embedding is a cheap fix: store a hash of each chunk's text alongside its vector, and only re-embed when the hash changes. This alone eliminates the most common source of wasted embedding spend without touching anything else in the pipeline.

Vector database cost usually scales with the number of vectors stored, their dimensionality, and how many replicas you run for availability. Reducing embedding dimensionality, some models offer a smaller output size at a modest quality cost, or quantizing vectors to int8 or binary representations can cut storage and memory footprint substantially, since the index has to fit in memory for fast search on most engines. This is worth testing against your recall benchmark before committing to it broadly.

Replica count is worth revisiting on its own schedule too. A team that added a third replica during an early scaling scare sometimes never revisits whether it's still needed once traffic patterns settle down, and it's a fixed cost that compounds every month it goes unreviewed.

Say your pipeline retrieves the top 20 chunks and reranks them before passing the top 5 to the model. Each of those steps has a per-call or per-token cost that's usually far higher than the vector search itself, which is typically a cheap, self-hosted or low-cost managed operation. If you're retrieving more chunks than you actually use in the final prompt, you're paying reranking cost for candidates that get discarded. Tightening your top-k before reranking is often the cheapest lever available.

It's worth pricing out the reranker separately from the generation model, since teams often bundle both into a single "the AI part is expensive" line item and never find out which one actually drives the bill. In many pipelines, generation, not reranking, is the larger cost, simply because the final prompt carries far more tokens than a reranking pass does.

If your chunking strategy produces overlapping or oversized chunks to be safe, you're paying generation-time token cost for redundant context on every single query, not just occasionally. Say a document is chunked with 500-token overlaps between adjacent chunks; retrieving five of those chunks means paying for roughly 2,000 tokens of duplicated content. Tightening chunk boundaries and overlap, and testing whether retrieval quality actually degrades without the extra padding, is worth doing before assuming you need a bigger context window.

Not all embedding models cost the same, and the price difference between a larger model and a smaller, cheaper one can be substantial at ingestion volume. The tradeoff is retrieval quality: a cheaper model with lower-dimensional embeddings can lose enough recall that your reranker, or your users, notice. Before switching, re-embed a representative sample of your corpus with the cheaper model and measure recall against your current baseline, not just against a generic published benchmark, since the quality gap varies a lot by domain and document type.

Say your ingestion pipeline embeds documents the moment they're uploaded, one at a time, through a real-time API call. Most embedding providers offer a batch endpoint at a meaningfully lower per-token price for workloads that can tolerate a delay of minutes to hours. If your source documents aren't time-sensitive, routing bulk or backfill ingestion through the batch path, while keeping real-time embedding only for genuinely urgent updates, can cut a meaningful share of ingestion cost without touching query-time behavior at all.

Work through these cost levers, starting with ingestion:

• Check ingestion logs for redundant re-embedding, and store a content hash per chunk so you only re-embed text that changed.
• Reduce vector dimensionality or quantize vectors to lower storage and memory cost, testing recall before you commit.
• Retrieve only as many chunks as the final prompt uses, since extra reranked chunks cost more per token than the search itself.
• Tighten chunk boundaries and overlaps so you do not pay for duplicated context on every generation call.
• Route bulk or backfill ingestion through a provider's batch endpoint when the documents are not time-sensitive.