CI/CD for Firmware When a Bad Build Reaches a Physical Machine

A web application's test suite runs against the same environment it will deploy to. Firmware's test suite usually can't, because a standard CI runner doesn't have the physical hardware the firmware controls. Your pipeline needs a layer standard software testing doesn't: either a simulator that models the hardware's behavior closely enough to catch most bugs, or an actual hardware-in-the-loop rig the pipeline can drive directly.

Both GitHub Actions and GitLab CI can trigger a build and hand it to either kind of test layer; neither platform includes hardware simulation itself, so this is infrastructure you're building on top of the CI platform, not something the platform gives you out of the box.

A firmware build that compiles cleanly and passes a simulator can still fail on real hardware because of a timing issue, a sensor calibration difference, or a component tolerance the simulator doesn't model precisely. Route firmware builds through an actual hardware-in-the-loop test station as a required pipeline step before a build is approved for a physical unit, not as an optional check someone runs when they remember.

Set up a self-hosted runner physically near the test rig, since a hosted runner has no path to a piece of lab equipment sitting on your production floor.

Manufacturing environments typically separate operational technology, the network running production equipment, from the IT network running everything else, for good security reasons. A CI/CD runner that bridges the two, by living on the OT network but pulling code from a cloud-hosted CI platform, can quietly undo that separation if it's not configured carefully.

Work with whoever owns your OT security policy before placing a self-hosted runner anywhere near production equipment, and keep the connection between the runner and the CI platform as narrow and outbound-only as the platform allows.

When a physical unit ships with a specific firmware version, you need a durable record connecting that unit's serial number to the exact build, commit, and test results that firmware went through. Build this into your pipeline as an automated step, writing the build's identifying information to a manufacturing record system rather than relying on someone manually logging it after the fact.

This matters most when a firmware issue surfaces months after shipment: without that record, tracing which units are affected becomes a much slower and more uncertain process than it needs to be.

Before a firmware build is approved to ship to a physical unit, confirm:

• Has the build passed hardware-in-the-loop testing on the actual test rig, not just a simulator?
• Is the build's version tied to a specific commit and test result set in your manufacturing record system?
• Does the runner that built and tested it maintain proper separation from your OT network?
• Is there a tested field-update or rollback path if an issue surfaces after units have already shipped?

A firmware pipeline's whole job is to prevent exactly the kind of failure that's expensive and slow to fix once it's out in physical units. Treat every one of these checks as non-negotiable rather than something to skip under a deadline.

Rolling back a web application means redeploying a previous version to a server you control. Rolling back firmware means pushing an update to units that may already be running in a customer's facility, on a network you don't control, sometimes with a person needing to be physically present. Test your field-update mechanism itself as part of your pipeline validation, not just the firmware build it delivers.

A firmware pipeline that produces perfect builds is only half the story if the mechanism for getting a fix onto already-deployed units is untested and unreliable when you actually need it. Include at least one full dry run of the update mechanism against a non-production unit in your regular release checklist.