Skip to content
Documentation Background

VLANs and overlays

A plain switched network is one broadcast domain. Every ARP request and DHCP discover reaches every host, and anyone plugged in can try to talk to anything. Two techniques split that up. VLANs divide one physical switch network into several logical ones. Overlays such as VXLAN carry a layer 2 network across a layer 3 one.


A VLAN is a separate broadcast domain defined in software. In the common port-based form, an administrator assigns each switch port to a VLAN, and the switch only forwards frames between ports of the same VLAN. Hosts in one VLAN behave as if they had their own private switch.

This fixes three problems of a flat network:

ProblemHow VLANs help
No traffic isolation. Broadcasts and unknown-destination floods reach everyone, and hosts can sniff each otherEach VLAN is its own broadcast domain
Poor use of switches. Separating ten small groups would need ten switchesOne switch hosts all ten VLANs
Moving a user means recablingReassign the port in software

Other membership schemes exist, based on the device’s MAC address or on the protocol in the frame. Port-based is by far the most common. In cloud and container platforms the idea reappears as network segments, subnets, and namespaces.

When a VLAN spans several switches, one cable per VLAN between switches would not scale. Instead a single trunk link carries all VLANs. To tell frames apart, the sending switch inserts a 4-byte 802.1Q tag after the source MAC address, and the receiving switch reads it and removes it before delivering to an access port.

Tag fieldSizeMeaning
TPID16 bitsFixed value 0x8100, marks the frame as tagged
Priority3 bitsClass of service for QoS
DEI1 bitWhether the frame may be dropped under congestion
VLAN ID12 bitsThe VLAN, 1 to 4094 usable

The ports connect hosts in access mode (no tag, one VLAN) or switches and servers in trunk mode (tagged, many VLANs). A frame without a tag on a trunk belongs to the native VLAN. Mismatched native VLANs between two switches leak traffic between segments, and trunk ports that accept every VLAN by default are an avoidable exposure. Prune trunks to the VLANs they actually need. See Network hardening.

VLANs are isolated at layer 2, so traffic between them must pass through a layer 3 device. Each VLAN is normally its own IP subnet, and the router has an interface (or sub-interface) in each. There are two common ways to attach it:

  • Router on a stick. One physical link from a switch trunk to a router, with a tagged sub-interface per VLAN. Simple, but all inter-VLAN traffic shares that link.
  • Layer 3 switch. The switch has a virtual interface (SVI) for each VLAN and routes internally at hardware speed. This is the usual design in an office or data center.
Terminal window
# Linux: create a VLAN sub-interface for VLAN 20 on eth0
sudo ip link add link eth0 name eth0.20 type vlan id 20
sudo ip addr add 10.20.0.1/24 dev eth0.20
sudo ip link set eth0.20 up
# Show VLAN details, and the VLANs on a Linux bridge
ip -d link show eth0.20
bridge vlan show

A common debugging pattern: a host gets the wrong DHCP lease or cannot reach its gateway, and the cause is the switch port in the wrong VLAN. Check show vlan brief (Cisco-style) or bridge vlan show first.


802.1Q has two limits. The 12-bit ID gives at most 4,094 usable VLANs, which is too few for a cloud provider with thousands of tenants. And a VLAN lives inside one switched layer 2 domain, which does not scale across routed data center fabrics or between sites.

VXLAN (RFC 7348) solves both by tunneling Ethernet frames inside UDP packets. Two hosts on different layer 3 networks can then share one layer 2 segment. The pieces:

  • VTEP (VXLAN tunnel endpoint): a switch, a router, or a software switch on a hypervisor. It wraps frames going into the tunnel and unwraps frames coming out.
  • VNI (VXLAN network identifier): a 24-bit ID, giving about 16 million segments.
flowchart LR
    HA["Host A"] -->|"Ethernet frame"| V1["VTEP 1<br/>10.0.0.1"]
    V1 -->|"Outer IP + UDP + VXLAN header(VNI 5000)<br/>wrapping the original frame"| NET["Routed IP network"]
    NET --> V2["VTEP 2<br/>10.0.1.1"]
    V2 -->|"Original frame"| HB["Host B"]

The routers in the middle forward ordinary IP/UDP packets between the VTEPs and never see the inner frame. The inner frame arrives unchanged, with the original source and destination MAC addresses, so Host A and Host B behave as if they were on the same wire.

802.1Q VLANVXLAN
Identifier12 bits (4,094 usable)24 bits (about 16 million)
Carried overOne layer 2 domainAny IP network
Method4-byte tag in the Ethernet headerEntire frame inside UDP/IP
Typical placeOffice and campus LANsCloud and data center fabrics
Terminal window
# Linux: a VXLAN interface with VNI 5000 to a remote VTEP (UDP port 4789)
sudo ip link add vxlan5000 type vxlan id 5000 \
dstport 4789 local 10.0.0.1 remote 10.0.1.1 dev eth0
sudo ip link set vxlan5000 up
# Attach it to a bridge so local workloads join the segment
sudo ip link set vxlan5000 master br0
  • MTU. The wrapper adds 50 bytes: 20 for the outer IP header, 8 for UDP, 8 for VXLAN, and 14 for the inner Ethernet header. A 1,500-byte underlay MTU leaves 1,450 for the overlay payload. Either lower the MTU inside the overlay or raise the underlay to jumbo frames. Forgetting this gives the classic symptom of small requests working and large transfers hanging, covered in fragmentation and PMTU.
  • Learning and flooding. Plain VXLAN floods unknown destinations to every VTEP, like a switch does. Production fabrics normally distribute the MAC and IP reachability with a control plane such as BGP EVPN, so VTEPs learn endpoints instead of flooding.
  • Visibility. Packet captures on the physical NIC show UDP between VTEPs. To see the workload traffic, capture on the VXLAN interface or decode UDP 4789.
  • Kubernetes. Several CNI plugins use VXLAN to connect pod networks across nodes. See Kubernetes networking model.