VLANs and overlays
A plain switched network is one broadcast domain. Every ARP request and DHCP discover reaches every host, and anyone plugged in can try to talk to anything. Two techniques split that up. VLANs divide one physical switch network into several logical ones. Overlays such as VXLAN carry a layer 2 network across a layer 3 one.
A VLAN is a separate broadcast domain defined in software. In the common port-based form, an administrator assigns each switch port to a VLAN, and the switch only forwards frames between ports of the same VLAN. Hosts in one VLAN behave as if they had their own private switch.
This fixes three problems of a flat network:
| Problem | How VLANs help |
|---|---|
| No traffic isolation. Broadcasts and unknown-destination floods reach everyone, and hosts can sniff each other | Each VLAN is its own broadcast domain |
| Poor use of switches. Separating ten small groups would need ten switches | One switch hosts all ten VLANs |
| Moving a user means recabling | Reassign the port in software |
Other membership schemes exist, based on the device’s MAC address or on the protocol in the frame. Port-based is by far the most common. In cloud and container platforms the idea reappears as network segments, subnets, and namespaces.
Trunks and the 802.1Q tag
Section titled “Trunks and the 802.1Q tag”When a VLAN spans several switches, one cable per VLAN between switches would not scale. Instead a single trunk link carries all VLANs. To tell frames apart, the sending switch inserts a 4-byte 802.1Q tag after the source MAC address, and the receiving switch reads it and removes it before delivering to an access port.
| Tag field | Size | Meaning |
|---|---|---|
| TPID | 16 bits | Fixed value 0x8100, marks the frame as tagged |
| Priority | 3 bits | Class of service for QoS |
| DEI | 1 bit | Whether the frame may be dropped under congestion |
| VLAN ID | 12 bits | The VLAN, 1 to 4094 usable |
The ports connect hosts in access mode (no tag, one VLAN) or switches and servers in trunk mode (tagged, many VLANs). A frame without a tag on a trunk belongs to the native VLAN. Mismatched native VLANs between two switches leak traffic between segments, and trunk ports that accept every VLAN by default are an avoidable exposure. Prune trunks to the VLANs they actually need. See Network hardening.
Routing between VLANs
Section titled “Routing between VLANs”VLANs are isolated at layer 2, so traffic between them must pass through a layer 3 device. Each VLAN is normally its own IP subnet, and the router has an interface (or sub-interface) in each. There are two common ways to attach it:
- Router on a stick. One physical link from a switch trunk to a router, with a tagged sub-interface per VLAN. Simple, but all inter-VLAN traffic shares that link.
- Layer 3 switch. The switch has a virtual interface (SVI) for each VLAN and routes internally at hardware speed. This is the usual design in an office or data center.
# Linux: create a VLAN sub-interface for VLAN 20 on eth0sudo ip link add link eth0 name eth0.20 type vlan id 20sudo ip addr add 10.20.0.1/24 dev eth0.20sudo ip link set eth0.20 up
# Show VLAN details, and the VLANs on a Linux bridgeip -d link show eth0.20bridge vlan showA common debugging pattern: a host gets the wrong DHCP lease or cannot reach its gateway, and the cause is the switch port in the wrong VLAN. Check show vlan brief (Cisco-style) or bridge vlan show first.
802.1Q has two limits. The 12-bit ID gives at most 4,094 usable VLANs, which is too few for a cloud provider with thousands of tenants. And a VLAN lives inside one switched layer 2 domain, which does not scale across routed data center fabrics or between sites.
VXLAN (RFC 7348) solves both by tunneling Ethernet frames inside UDP packets. Two hosts on different layer 3 networks can then share one layer 2 segment. The pieces:
- VTEP (VXLAN tunnel endpoint): a switch, a router, or a software switch on a hypervisor. It wraps frames going into the tunnel and unwraps frames coming out.
- VNI (VXLAN network identifier): a 24-bit ID, giving about 16 million segments.
flowchart LR
HA["Host A"] -->|"Ethernet frame"| V1["VTEP 1<br/>10.0.0.1"]
V1 -->|"Outer IP + UDP + VXLAN header(VNI 5000)<br/>wrapping the original frame"| NET["Routed IP network"]
NET --> V2["VTEP 2<br/>10.0.1.1"]
V2 -->|"Original frame"| HB["Host B"]
The routers in the middle forward ordinary IP/UDP packets between the VTEPs and never see the inner frame. The inner frame arrives unchanged, with the original source and destination MAC addresses, so Host A and Host B behave as if they were on the same wire.
| 802.1Q VLAN | VXLAN | |
|---|---|---|
| Identifier | 12 bits (4,094 usable) | 24 bits (about 16 million) |
| Carried over | One layer 2 domain | Any IP network |
| Method | 4-byte tag in the Ethernet header | Entire frame inside UDP/IP |
| Typical place | Office and campus LANs | Cloud and data center fabrics |
# Linux: a VXLAN interface with VNI 5000 to a remote VTEP (UDP port 4789)sudo ip link add vxlan5000 type vxlan id 5000 \ dstport 4789 local 10.0.0.1 remote 10.0.1.1 dev eth0sudo ip link set vxlan5000 up# Attach it to a bridge so local workloads join the segmentsudo ip link set vxlan5000 master br0Costs and caveats
Section titled “Costs and caveats”- MTU. The wrapper adds 50 bytes: 20 for the outer IP header, 8 for UDP, 8 for VXLAN, and 14 for the inner Ethernet header. A 1,500-byte underlay MTU leaves 1,450 for the overlay payload. Either lower the MTU inside the overlay or raise the underlay to jumbo frames. Forgetting this gives the classic symptom of small requests working and large transfers hanging, covered in fragmentation and PMTU.
- Learning and flooding. Plain VXLAN floods unknown destinations to every VTEP, like a switch does. Production fabrics normally distribute the MAC and IP reachability with a control plane such as BGP EVPN, so VTEPs learn endpoints instead of flooding.
- Visibility. Packet captures on the physical NIC show UDP between VTEPs. To see the workload traffic, capture on the VXLAN interface or decode UDP 4789.
- Kubernetes. Several CNI plugins use VXLAN to connect pod networks across nodes. See Kubernetes networking model.