Why Your Test Suite Passes and Your Deploys Still Break Things

Mocked dependencies in a unit test suite are only as accurate as whoever wrote the mock, and mocks drift from the real service's behavior the moment that service changes without the mock being updated to match. A test suite that's fully green can still be testing against a contract that no longer exists.

Integration tests against a real, or realistically faked, version of the dependency catch this class of drift. They're slower and more expensive to run, which is exactly why most teams under-invest in them relative to unit tests, but they catch the failures unit tests structurally can't.

Deploying a new version to one hundred percent of traffic at once means a bad change reaches every user simultaneously, and rollback happens after the damage, not before it. A canary or staged rollout, five percent of traffic, then twenty five, then everyone, catches most regressions while they're still affecting a small slice.

This needs real metrics watching the canary automatically, error rate, latency, not just a person eyeballing a dashboard for ten minutes before moving on. Staged rollouts are also what make a high deploy frequency safe in the first place: benchmark data on deploy frequency shows the fastest-moving engineering organizations shipping on demand, essentially daily, versus roughly monthly at the slowest end, and getting there depends on each individual release being small and cheap to verify¹.

A rollback path that's never been exercised outside an emergency is a rollback path you can't trust during an emergency. Practicing a rollback on a low-stakes service, on a schedule, not just when something's actually broken, is what turns 'we can roll back' from a hope into a verified fact.

This matters more in a distributed system than a monolith, because rolling back one service while its dependencies have already moved forward can itself cause a compatibility break. Rollback plans need to account for what else changed since the version being restored.

Say a service's unit tests all pass after a schema change to an event it publishes, because the tests only check the publisher's own behavior. A downstream consumer, tested separately with a mock of the old event shape, breaks the moment the new version reaches production and the real event doesn't match what the mock promised.

A contract test between publisher and consumer, run in the pipeline before either side deploys, would have caught this before it shipped. This is the specific gap between 'tests pass' and 'the system works together' that services tested in isolation can't close on their own.

• Integration tests skipped because they're slow, so only unit tests gate the deploy
• A staged rollout with no automated rollback trigger, relying on someone noticing the dashboard
• Rollback tested for the first time during an actual incident
• Contract tests that exist for some service pairs and not others, usually the newest ones

A pipeline that only checks unit tests is doing part of the job and giving false confidence about the rest. Closing that gap doesn't mean adding every possible check at once, it means identifying which failure modes have actually hit you in production and building the specific check that would have caught each one.

The pipeline earns trust the same way a person does: by catching real problems over time, not by being exhaustive on paper. Start with the last two or three incidents that a green pipeline didn't prevent, and build backward from there.