A Production Deployment Checklist That Actually Catches Problems

A rollback plan that has never been tested is a guess. Before shipping anything that touches a shared dependency, database schema, or message format, confirm you can revert the code without reverting the data, or that reverting the data is also safe.

Database migrations are the usual trap: a migration that drops a column can't be rolled back by redeploying the old code, because the old code still expects that column to exist. Expand-then-contract migrations, where you add the new shape, deploy code that works with both, then remove the old shape later, avoid this.

In a distributed system, deploying the consumer of an API before the producer supports the new contract breaks requests immediately. Deploying the producer first and keeping it backward compatible until every consumer has updated is the safer default.

Write down the dependency order for any change that touches more than one service, and don't rely on remembering it under pressure during a release window.

A canary release to a small slice of traffic, watched against your key metrics before a full rollout, catches problems that pass every test but show up under real traffic patterns, uneven data, unusual request shapes, load that a staging environment never replicates.

Deploy frequency is a real signal of engineering health, and teams that release small, frequent changes tend to ship faster and with less risk per release than teams batching everything into a monthly window: research on deploy frequency puts the fastest-moving teams shipping on demand, essentially daily, while the slowest cluster manages roughly one release a month¹. A smaller, more frequent release is also just easier to reason about: fewer changes bundled together means a regression points to a shorter list of suspects.

• Verifying the feature flag actually defaults to off before the code ships
• Checking that the rollback path has been exercised in staging, not just written down
• Confirming downstream consumers know about a contract change before it goes out
• Watching error rates and latency for at least one full traffic cycle after the deploy, not just the first five minutes

Every one of these takes minutes to do right and hours to recover from if skipped.

Decide in advance who has the authority to roll back without a meeting. The minutes spent debating whether a spike is real or noise are usually the most expensive minutes of an incident.

Write a short postmortem for every rollback, even the quiet ones that never became customer-facing incidents. The pattern that caused a near-miss this month is often the same one that causes a real incident next quarter if it goes unaddressed.

In a distributed system with several teams, the riskiest deploys aren't the big ones everyone plans around, they're the small ones that quietly change a contract another team depends on. A release calendar that other teams can actually see, not just your own, cuts down on the surprise factor.

For any change to a shared API or message format, give downstream teams a backward-compatibility window: ship the new shape alongside the old one, announce a removal date, and only take the old shape away once you've confirmed nobody's still calling it. Skipping that window is the single most common cause of a deploy that looks clean in isolation but breaks something two services away.

Say the payments team renames a field in an event it publishes. If three other services consume that event and only one of them is on the release calendar you checked, the other two start failing silently the moment the change ships, and nobody notices until a downstream report comes back wrong days later. A quick search across the codebase for who subscribes to an event, done before the change goes out, catches this in minutes instead of after the fact.