Zero Trust for a RAG Pipeline Means No Service Gets a Free Pass

A user request gets verified with something like a session token tied to an identity and a permission set. An internal service call needs the equivalent: a service identity, not a shared network location, and a scoped credential that proves this specific service is allowed to make this specific call, not just that the call originated somewhere on the internal network. If your retriever can query the vector database using the same broad credential your ingestion pipeline uses, you don't have service-level verification, you have one shared secret with two names.

The test for zero trust isn't whether each service is individually secure. It's what happens if one is compromised anyway: can a compromised reranking service query the vector database directly for documents it was never supposed to see, or call the generation model with an arbitrary prompt? If the answer is yes, the components are on the same trust boundary even if they're logically separate services, and zero trust hasn't actually been applied between them.

The retriever needs read access to the vector database, not write access. The ingestion pipeline needs write access to add new documents, not the ability to query on behalf of a user. Each service in the pipeline should hold a credential scoped to exactly its job, so a compromise of one service limits an attacker to what that service was allowed to do, not to the full range of what any service in the pipeline can do.

For example, a retriever and an ingestion job both connect to the vector database with the same admin key. A flaw in the retriever's dependencies lets an attacker run arbitrary calls, and because the key can write and delete, they can modify or remove indexed documents, not just read them. Splitting the credentials gives the retriever a read-only key and the ingestion job a write key held only where ingestion runs. The same compromise now exposes read access at most, and a denied write attempt shows up in the logs instead of succeeding silently.

Every verification check adds latency, and a RAG pipeline is already latency-sensitive across several hops. This is a real tradeoff, not a reason to skip verification: use short-lived, cacheable credentials rather than a full identity check on every single call, so you get the security benefit without a network round trip added to every hop in your response-time budget.

• Does each internal service call carry its own scoped credential, or does it rely on network location alone?
• Is the retriever's access to the vector database read-only, separate from the ingestion pipeline's write access?
• If one service in the pipeline were compromised, what else could it reach?
• Are verification checks cached or short-lived enough not to blow the response-time budget?
• Does a credential leak from one service expose more than that service's own scope?

Few teams get a fully verified, scoped, least-privilege pipeline on the first pass, and that's fine as a starting point rather than a finish line. Rank your hops by what a compromise there would actually expose, the vector database holding your full corpus is a higher-value target than a stateless reranker, and fix the highest-value gaps first instead of trying to verify every hop equally in one project.

A scoped credential that quietly blocks a call it shouldn't have made is doing its job, but only if someone can see it happened. Log every denied service-to-service call with enough detail to tell whether it was a misconfiguration on your side or an actual compromise attempt, and review that log regularly. Without it, a service that's been silently probing for access it doesn't have looks identical to a service that's working correctly, right up until the probing succeeds.