Setting Up VPC Peering Around a Streaming Cluster
Isolate a streaming cluster by placing it in its own private subnet, peering only the subnets that hold producers and consumers, and denying all other traffic by default. A single broker often has read access to data from several parts of the business, so reaching it should require more than sitting in the same cloud account.
The mechanics, VPC peering, private subnets, and security groups, are well understood. The mistakes that actually cause incidents are almost always about what gets left open by default, not about the isolation technique itself.
Put the cluster in its own private subnet, not a shared one
Brokers and stream processors should live in a subnet with no direct route to the public internet, reachable only through peering connections or a private link from the services that actually need them. This sounds obvious, but clusters started quickly during an early build-out often land in a general-purpose subnet alongside everything else, and nobody circles back to move them once traffic is flowing.
Moving an established cluster to a new subnet later is disruptive, so this is worth getting right at setup even if it means a slower initial rollout.
How narrowly should you peer VPCs around a streaming cluster?
A peering connection between two VPCs does not have to expose every subnet on both sides. Scope route tables so that only the specific subnets containing producers and consumers can reach the cluster's subnet, rather than peering the entire network and relying on security groups alone to narrow it down afterward.
This matters because security groups are easy to loosen under pressure during an incident and easy to forget to tighten back up afterward, while a route table that was never opened in the first place does not depend on anyone remembering to close it.
Deny by default and log every rejected connection
Set security group and network ACL defaults to deny, then add explicit allow rules for known producers, consumers, and admin access paths. This inverts the more common but riskier pattern of starting open and trying to remember to close things down as the cluster matures.
Route rejected connection attempts to a log you actually look at, not just one that exists. A steady trickle of denied connections from an unexpected source is often the first visible sign of a misconfigured service or, less often, something worse, and it is only useful as a signal if someone is watching it.
How do you test that the isolation actually works?
The most common isolation failure is not a missing rule, it is a rule that is more permissive than whoever wrote it believed. Periodically test reachability from a host that should not have access and confirm it is actually blocked, rather than trusting the configuration as written.
A simple version of this: spin up a temporary instance in an unrelated subnet and try to reach the cluster's broker port directly. If that succeeds, you have found a real gap before an attacker or a misrouted internal service does.
Route admin access through a bastion, not a direct rule
Engineers who need to reach the cluster for debugging or maintenance should go through a bastion host or a session manager tool with its own logging, not through a standing security group rule that lets their laptop's IP address reach the broker port directly. Direct rules like that tend to accumulate, since removing someone's access when they change teams is a task that is easy to forget compared with adding it in the first place.
A bastion also gives you one place to look when you need to know who actually connected to the cluster and when, instead of piecing that answer together from a list of security group entries that only shows what was allowed, not what happened. Rotate and review bastion access on the same recurring schedule you use for reviewing who can deploy to production, rather than treating it as a separate, lower-priority list.
A quick isolation checklist for the cluster:
- Security group and network ACL defaults are set to deny, with explicit allow rules only for known producers, consumers, and admin paths.
- Route tables expose only the subnets that contain producers and consumers, not the entire peered network.
- Rejected connection attempts flow into a log that someone actually reviews on a regular schedule.
- Admin access goes through a bastion host or session manager with its own logging, not standing rules for individual laptops.
- Reachability from an unrelated subnet is tested periodically and confirmed to be blocked.
What Good Looks Like
A properly isolated streaming cluster sits in its own private subnet, is reachable only through narrowly scoped peering connections, denies by default with logged rejections, and gets its isolation tested periodically instead of trusted on paper.
Building The Capability (5-Stage Skill Ladder)
How to Get Started
Frequently Asked Questions
Do we need a separate VPC for a streaming cluster, or is a private subnet enough?
A private subnet with tightly scoped peering and security groups is usually enough for most teams. A fully separate VPC adds another layer of isolation and makes sense once you have compliance requirements or multiple teams that should not share network access to the same cluster.
How often should we test that our network isolation actually works?
A quarterly reachability test from an unrelated subnet is a reasonable baseline, and it is worth repeating after any change to peering connections or route tables, since that is when a rule most often ends up broader than intended.
What is the most common network isolation mistake with streaming clusters?
Peering an entire VPC instead of scoping route tables to just the subnets that need access. It works fine until a service that has no business reaching the cluster technically can, because nothing was actually stopping it beyond a security group someone might loosen later.
About the numbers
This guide doesn't quote a sourced benchmark. Figures in it are estimates or general guidance, so check them against your own numbers.
Related Guides
Where VPC Peering Breaks Down and How to Isolate Blast Radius Instead
Why VPC peering alone doesn't isolate anything, and a more reliable way to contain blast radius between services and environments.
VPC Peering Looks Like Isolation Until You Check the Routes
VPC peering can silently become transitive, undoing the isolation you thought you had. Here is how to audit what can actually reach what in your network.
What a Misconfigured VPC Peering Connection Actually Breaks
A walkthrough of a real VPC peering misconfiguration, what it exposed, and the four checks that would have caught it before it shipped.
The VPC Peering Mistake That Opens Your Whole Network
How VPC peering misconfigurations quietly expose more of your network than intended, and four concrete checks that catch the mistake before an audit does.
A Practical Checklist for VPC Peering and Network Isolation
The specific network isolation mistakes that quietly undermine a zero-trust architecture, and a checklist for catching them in your VPC peering setup.
Four Network Isolation Checks Most VPC Peering Setups Skip
Four specific checks for VPC peering and network isolation setups, plus the pitfalls that let a segmentation boundary look correct while quietly failing.