The IaC Setup That Works Until Someone Changes Something by Hand

One manual change during an incident is understandable and sometimes necessary. The problem is what happens next: nobody updates the code to match, the next person who runs the IaC either reverts the fix unknowingly or the tool reports a change it doesn't actually understand, and now there's uncertainty about what the code will do the next time it runs. Left unaddressed, drift accumulates until the code is no longer a reliable description of what's actually running.

A simple rule keeps one-time exceptions from becoming ongoing drift: any manual console change made during an incident gets a follow-up ticket before the incident is closed. The ticket names who will update the code, and by when. For example, if an engineer opens a security group to restore access at night, the morning handoff includes a task to encode that rule in the code or remove it. This costs a few minutes per incident, and it means the next full reapply is a routine event rather than a surprise that quietly reverts something still in use.

Run a drift detection check on a recurring schedule, comparing what the code declares against what's actually deployed, rather than discovering drift the hard way when an unrelated change to the same resource produces an unexpected result. Most IaC tools support a plan or diff mode that surfaces this without applying anything. Treat any detected drift as something to investigate and resolve deliberately, either by updating the code to match reality or reverting the manual change, not something to silently accept.

A repeatable routine for closing drift looks like this:

1. Run a scheduled plan or diff that compares what the code declares against what is actually deployed, without applying anything.
2. Triage each difference by what it touches, so a production firewall rule gets attention before a development resource tag.
3. Resolve each item deliberately, either by updating the code to match reality or by reverting the manual change.
4. Before closing any incident that involved a console change, record it and update the code so the exception does not become permanent.

The state file tracking what the tool believes is deployed can itself become corrupted, out of sync, or locked by a crashed process mid-run, and a state file problem can be more disruptive than the infrastructure issue that triggered the original manual change. Back up state before risky operations, use remote state with locking to prevent two people running changes simultaneously, and know the recovery procedure for a corrupted or lost state file before you need it during an actual incident.

A monolithic configuration where every resource lives in one giant file means a small, well-intentioned change carries the blast radius of the entire infrastructure. Breaking configuration into smaller, focused modules, networking separate from compute separate from data stores, limits how much any single change can affect, and makes it possible to review and approve changes to one area without touching everything else at once.

A 99.95% uptime target leaves roughly 4.38 hours a year of downtime budget across every incident combined¹, which is a useful anchor for how much review a given infrastructure change deserves: a change to a production database's configuration warrants more scrutiny than a change to a development environment's logging settings, and the review process should reflect that difference explicitly rather than treating every pull request the same.

Say a security group was manually adjusted during an incident to allow emergency access, and the code was never updated to match. Six months later, an unrelated infrastructure change triggers a full reapply, which silently reverts that manual adjustment because the code never knew about it. If that manual change had actually been load-bearing for something still running, the rollback becomes a new incident, caused entirely by drift nobody tracked down after the original fix.

Not all drift carries the same risk: a manually adjusted tag on a development resource is very different from a manually adjusted firewall rule on a production database, and treating both with the same urgency either wastes time chasing harmless drift or, worse, lets genuinely risky drift sit unresolved because it's buried in a long list of low-stakes items. Triage detected drift by what it actually touches before deciding how fast it needs to be reconciled.