Build or Buy: Canary Deployments for a Small Team

A canary is only useful if you're watching the right signal on the slice of traffic hitting it: error rate, latency, and any business metric that would flag a subtler problem, like a checkout completion rate that quietly drops without throwing a single error. Watching only server error codes misses the class of bug that degrades the product without technically failing.

Decide the promotion criteria before the first canary ships, not while staring at a dashboard mid-rollout. A vague sense that things look fine is how a bad deploy gets waved through under pressure.

Most load balancers and ingress controllers already support weighted routing between two versions. Standing up a canary this way, ten percent of traffic to the new version, ninety to the old, for example, is a configuration change, not a new system. For a small team, this covers a surprising amount of what a canary needs to do.

What it doesn't give you is automatic analysis. Someone has to watch the metrics and decide, by hand, whether to promote or roll back. That's a real cost, but it's a cost you can absorb manually long before it's worth automating.

For example, a small team can run a weighted split of ten percent to the new version, then follow a written checklist: check error rate, latency, and one business metric at fixed intervals, promote only if all three match the old version, and roll back as soon as any of them diverges. The checklist turns a vague sense that things look fine into a decision anyone on call can make. When the manual steps start to eat real engineering time, that is the signal to consider automating them.

A dedicated progressive delivery platform automates the part a manual canary leaves to a human: it watches the metrics you define, shifts traffic upward on its own when they look healthy, and rolls back automatically when they don't. A canary rollout is what makes on-demand deployment survivable instead of risky, which is exactly the habit that separates DORA's highest-performing teams from everyone else¹.

That automation is worth paying for once deploys are frequent enough that a person watching every one by hand becomes the bottleneck, not before.

Error rate and latency are the obvious pair, but they're not sufficient on their own. Add at least one metric tied to what the feature is actually supposed to do, so a canary that's technically healthy but functionally broken still gets caught. And set a minimum observation window, because a canary that looks fine for ninety seconds can still be hiding a problem that only shows up under sustained load.

Write these thresholds into the deploy process itself where you can, even if enforcement is still manual at first. A checklist that says what green and red look like for this specific release is worth more under pressure than a person's memory of what normal usually looks like.

A written promotion rule should cover these points:

• Error rate and latency on the slice of traffic that hits the canary.
• At least one metric tied to what the feature is supposed to do, so a technically healthy but functionally broken canary is still caught.
• A minimum observation window, since a canary that looks fine for ninety seconds can still hide a problem that appears under sustained load.
• What green and red look like for this specific release, written down before the rollout starts.

Start with manual, weighted traffic shifting on infrastructure you already run. Move to a managed platform once deploys are frequent enough that manual promotion decisions are eating real engineering time, or once a bad deploy has actually reached full traffic because nobody was watching closely enough during the canary window. That second trigger is the one that tends to make the decision for you.

There's a middle option worth knowing about too: some teams write a small script that polls their existing metrics platform and pages someone the moment a canary's error rate crosses a threshold, well short of building full automated promotion. That covers the worst failure mode, a bad deploy running unnoticed, without the cost of a full platform.