Every time I explain VPCs in VCF 9.1, someone asks the same thing: ‘where does the packet actually go?’ So I drew it out, from the VCF Automation namespace all the way down to the vmnic and the ToR. There may already be plenty of posts like this, but I decided to put everything together and describe it anyway. Maybe it will be useful to someone. Any comments or suggestions are welcome:)
The result is three diagrams. A logical view, a physical view and a tenancy map. And below you can finda short walk through each one.
1. Logical view
[ logical view]
I used two NSX projects side by side on purpose, because they show the two connectivity models VCF 9.1 gives you.
- Project A uses a Distributed Transit Gateway (DTGW). No Edge, no T0, no BGP. The TGW connects straight to a VLAN that is available on every host. It holds VPC1 and VPC2.
- Project B uses a Centralized Transit Gateway (CTGW). Traffic goes through the Edge cluster and the T0. It holds VPC3.
Each VPC has its own VPC gateway, which is really a distributed router (DR) running on every host. On top of that i drew four packet flows:
- A (red), same VPC: VM1 → VPC1 GW → VM2. Routed locally, the TGW never sees it.
- B (orange), VPC to VPC: VM1 → VPC1 GW → TGW-A → VPC2 GW → VM3.
- C (blue), north-south with DTGW: VM1 → VPC1 GW → TGW-A → external VLAN → ToR. The dashed C* path is the new part in 9.1: if you need NAT, load balancing or a stateful firewall, the packet hairpins through the VPC service router on a VNA (Virtual Network Appliance) node first. In VCF 9.0 you basically had to go centralized for that.
- D (green), north-south with CTGW: VM4 → VPC3 GW → TGW-B → Edge (TGW SR → T0 SR) → BGP uplink.
2. Physical view
[physical view]
The second diagram shows the hosts and each host has 4 pNICs and two VDS:
- vds-mgmt (vmnic0/vmnic1): ESX management, vMotion, vSAN and the VM Management port group.
- vds-nsx (vmnic2/vmnic3): host TEPs, the DTGW VLAN segment, and on the Edge host the trunk port groups for the Edge interfaces.
vmnic0 and vmnic2 go to ToR-A, vmnic1 and vmnic3 to ToR-B. Host 3 also runs a VNA node and the Supervisor control plane VM, and the Edge host runs the NSX Edge with the TGW and T0 service routers.
A few things become obvious once you see it this way:
- All routing happens on the source host. VPC DR, TGW DR and the next VPC DR are all local. The only thing that crosses the wire between hosts is a Geneve packet from TEP to TEP.
- DTGW traffic is never tunnelled north-south. It leaves the host on the external VLAN through its own uplink and hits the gateway on the ToR. That’s why the VLAN has to be trunked to every host running VPC workloads.
- CTGW traffic always takes the extra hop. Host TEP → Edge TEP over Geneve, then the Edge sends it out its uplink VLAN to the router with BGP.
3. Networks and VLANs
VLAN IDs below are from my scenario, so swap in your own.
NOTE: VCF management services network is not visible in here but of course it’s a part of VCF 9.1 (seperated or shared with VM Management network)
Host TEP and Edge TEP are of course different VLANs that must be routed to each other, and the overlay needs at least 1700 MTU end to end ( i just use 9k).
4. Tenancy: who owns what
[tenancy mapping]
This section connects the network to VCF Automation, because that’s where people actually create things. It’s three columns (VCF Automation, NSX, vCenter/Supervisor) lined up so you can follow one workload across.
- Region: the Supervisors that share one NSX Manager. The provider hands each organization a region quota from it.
- All Apps organization = one NSX project. Setting up the org’s region networking creates its Transit Gateway, a default VPC, a default connectivity profile and outbound SNAT.
- A VCF Automation project is not an NSX project. The VCFA project is about users, roles and policies. One org can have many of them.
- Namespace → VPC. A namespace lives inside a VCFA project, gets its limits from the region quota and a namespace class, and is bound to a VPC from the org’s NSX project.
- Workloads → subnets. VM Service VMs and VKS clusters in the namespace get vNICs on that VPC’s subnets, which on the hosts are just the DRs and segments from the physical view.
In my example, Org A maps to NSX Project A (ns-a1 → VPC1, ns-a2 → VPC2) and Org B maps to NSX Project B (ns-b1 → VPC3).
