Why Your RAG Infrastructure Drifted From What Terraform Says It Should Be

During an incident, an engineer opens the cloud console and manually widens a network rule or bumps a vector database's instance size to resolve the immediate problem, then moves on once things are stable. Nobody backports that change into the Terraform definition, because the incident is over and there's a next thing to work on. Weeks later, someone runs a routine plan and it proposes reverting that manual fix back to the old, broken configuration, because as far as the code is concerned, the fix never happened.

Running a plan only when you're about to make an intentional change means drift sits undetected for however long it's been since the last deliberate change, which for a stable piece of infrastructure could be months. Run a drift detection job on a recurring schedule, independent of whether anyone's actively changing that infrastructure, so a manual change gets flagged and reconciled deliberately, either backported into code or reverted, instead of discovered by accident.

The core problem isn't that engineers make manual changes during incidents, it's reasonable to fix a production issue as fast as possible. It's that the fix doesn't make its way back into IaC afterward. Build incident response practice to include a follow-up step, opening a pull request to codify whatever manual change was made, as a required part of closing out the incident, not an optional cleanup task that competes with the next fire for attention.

For example, an on-call engineer resizes the vector database instance from the console late at night to stop query timeouts, and the incident ticket closes with a note describing what was done. If closing that ticket also requires a pull request that updates the definition, the next plan matches reality. If it does not, the next apply may shrink the instance back to its old size and bring the original outage back, which then looks like a mysterious regression rather than an unrecorded fix. Making the codifying step a required part of incident closure is what prevents that.

Vector database configuration, embedding inference container settings, and network rules for the RAG pipeline specifically sometimes get treated as less critical than core application infrastructure, especially when they were stood up quickly during an initial build. Apply the same IaC review process, pull request, approval, plan review before apply, to these resources as to anything else in production, since a misconfigured vector database or an overly permissive network rule here is just as real a risk as anywhere else.

Beyond the infrastructure itself drifting from its definition, the IaC state file can drift from reality too, if a resource is deleted outside of Terraform or Pulumi and the state file isn't updated to match, or if two people apply changes to overlapping infrastructure without coordinating and one apply silently clobbers the other's work. Use remote state with locking to prevent concurrent applies from colliding, and periodically reconcile state against actual cloud resources, not only definitions against actual resources.

• Is drift detection run on a recurring schedule, independent of active changes?
• Does incident response practice include a required follow-up step to codify manual fixes?
• Do RAG-specific resources, vector database, inference containers, get the same review rigor as core application infrastructure?
• Is remote state used with locking to prevent concurrent applies from colliding?
• Has anyone reconciled the state file against actual cloud resources recently, not just definitions against resources?

A drift detection job that emails its results to one person's inbox tends to produce the same outcome as no drift detection at all once that person is busy or out. Post findings somewhere the whole team sees them, and treat an open drift finding as something with the same visibility and urgency as an open bug, rather than a private notification that quietly ages unread.