A Network Isolation Checklist for a Production Inference Cluster

Every request should pass through a gateway or load balancer before it reaches a node actually running inference. Confirm this by trying to reach a GPU node's address directly from outside your network rather than trusting a diagram that says it is private. Security group and firewall rules drift over time as new services get added, and a rule opened temporarily for debugging is the most common way this quietly breaks.

If your model serving cluster peers with other networks, whether a data pipeline, a customer's own VPC, or another internal service, check that the peering connection only allows the specific ports and directions of traffic it needs, not open access between the two networks. A peering connection set up broadly to get an integration working quickly and never tightened afterward is a common source of unintended exposure a year later.

If one cluster serves multiple customers or business units, confirm that a request tagged for one tenant cannot reach another tenant's data or model configuration, whether through a shared cache, a shared queue, or a routing bug. Test this directly rather than relying on the intended design: send a request as one tenant and confirm nothing in the response or in logs reveals another tenant's information.

Isolation only matters if you would know when it fails. Confirm that connection attempts to your GPU nodes from outside their expected sources generate an alert someone actually sees, not just a log line nobody reads. A compliance automation platform like Vanta can help here by continuously checking your cloud network configuration for drift against the isolation rules you set, rather than relying on someone remembering to re-check manually.

The most common failure is not a bad initial design. It is drift: a rule opened for a debugging session and never closed, a new service added to the network without going through the same review as the original setup, or a peering connection expanded for one feature and never narrowed back down. Schedule a recheck of all four items above on a fixed calendar, not only when something prompts you to look.

Recheck these four items on a fixed calendar:

• Confirm GPU nodes cannot be reached directly from the public internet by testing from outside your network, not by trusting a diagram.
• Confirm each peering connection allows only the specific ports and directions of traffic it needs, and nothing broader.
• Confirm a request sent as one tenant cannot reveal data or configuration belonging to another tenant through a shared cache or queue.
• Confirm an unexpected connection attempt to a GPU node triggers an alert that a person actually sees and acts on.

Say an engineer opens a security group rule to reach a GPU node directly while chasing a hard-to-reproduce latency bug during an incident. The bug gets fixed, the incident closes, and the rule stays open because closing it was never part of the incident checklist. Six months later, a routine external scan finds the node reachable from outside the network, and nobody remembers why. None of the four checks above are hard to run. What makes them work is treating every exception opened during an incident as something with an expiration date, not a permanent fixture nobody circles back to close.

Write down what your isolation setup is supposed to look like: which subnets should be private, which peering connections exist and why, and what the expected alert behavior is for an unexpected connection attempt. Without that written baseline, a new engineer reviewing the network has no way to tell an intentional configuration from an accidental one, and drift becomes much harder to spot because nobody has a clear picture of what correct looks like.