Hardening the Containers Behind Your Model Serving Layer

Machine learning base images are often larger than necessary because they bundle build tools, compilers, and libraries only needed to build the image, not to run it. A smaller runtime-only image has a smaller attack surface and fewer packages that could carry a vulnerability. Separate your build stage from your runtime stage and ship only what the running container actually needs to serve inference. This also tends to speed up cold starts, since a smaller image pulls faster when a new node needs to come online during a scale-out event.

GPU workloads sometimes get built running as root because certain driver setups are easier to get working that way during initial development, and the shortcut quietly survives into production. Confirm your model serving containers run as a non-root user with the minimum permissions needed to access the GPU device, not more. This is a standard container security practice that gets skipped more often here specifically because GPU tooling has a reputation for being finicky about permissions.

A general container scanner may not have good coverage of machine learning specific libraries and their dependency chains, which move fast and sometimes carry vulnerabilities that a generic scanner's database has not caught up with yet. Confirm whatever scanning tool you use, such as Tenable, is actually flagging known issues in your specific ML library versions, not just your base operating system packages, by testing it against a library version you know has a published vulnerability.

For example, before trusting a scanner, build a throwaway image that pins an older ML library version with a published advisory and scan it. If the report lists only operating system packages and stays silent on the library, you have found a coverage gap. Record the result, then choose between adding a second scanner focused on language-level dependencies or asking your vendor how it covers ML libraries. Repeat the test whenever you change scanners or adopt a major new library, because a tool that caught the issue last quarter may not cover a newly added dependency.

GPU drivers are patched less frequently than most container security checklists assume, partly because a driver update can be riskier to roll out than an ordinary library patch, given how tightly coupled driver and hardware behavior can be. That is a reason to test driver updates carefully, not a reason to skip them. Put driver patching on an actual schedule with a defined testing step, rather than letting it happen only when a new instance type forces the issue.

A clean scan of the image at build time does not guarantee the running container stays clean, since a compromise can happen after deployment through a vulnerability in the running process itself. Endpoint monitoring tools such as CrowdStrike extend the check into runtime, watching for behavior that does not match what a model serving process should normally be doing, which catches a different class of problem than a build-time scan ever could.

Model weights baked into or mounted inside a container are often the single most valuable asset that container holds, sometimes more valuable than the application code around them. Treat access to that layer with the same care you would give a credentials file: restrict which processes can read it, avoid leaving it in a layer that gets cached and shared more broadly than intended, and confirm that a container escape would not hand an attacker a clean copy of weights representing real product investment. Weights mounted from external storage rather than baked into the image also make rotation and access revocation simpler, since revoking access to the storage location does not require rebuilding and redeploying every image that used it.

Run this checklist against each model serving image before it ships:

• Build and runtime stages are separate, and the final image contains only what the running container needs to serve inference.
• The container runs as a non-root user with the minimum permissions needed to reach the GPU device.
• Your scanner has been tested against a library version with a known published vulnerability, so you know it covers ML packages.
• GPU driver patching has a schedule and a defined testing step instead of waiting for a new instance type to force it.
• Model weights are mounted from external storage where possible, with access restricted to the processes that need them.