Data center networking
A data center network connects tens of thousands of servers inside a building and ties the building to the internet. The design goals differ from a campus network: very high bandwidth between servers, predictable latency, resilience to failure of any single component, and automation, since nobody configures thousands of switches by hand. This is the physical network under cloud instances and Kubernetes nodes.
Building blocks
Section titled “Building blocks”- Servers are commodity machines, stacked in racks of a few dozen.
- A top-of-rack (ToR) switch connects every server in its rack, usually over 25, 40, or 100 Gbit/s links. Traffic between two servers in the same rack goes through only this switch.
- Higher tiers of switches interconnect the racks.
- Border routers connect the data center to the internet and to other sites.
- Load balancers sit in front of the application servers. A layer 4 or layer 7 load balancer takes the public virtual IP for a service and spreads connections across backends, translating the address like NAT does. It also hides the internal addressing from clients.
Two directions of traffic:
| Direction | Between | Notes |
|---|---|---|
| North-south | Clients outside and servers inside | Goes through border routers and load balancers |
| East-west | Servers inside the data center | Service calls, storage, replication, distributed jobs. Usually the larger volume by far |
East-west dominates in microservice and big-data workloads, since one user request can fan out into dozens of internal calls. The network is designed around it.
From three tiers to leaf-spine
Section titled “From three tiers to leaf-spine”Three-tier hierarchy
Section titled “Three-tier hierarchy”The classic design has an access layer (ToR), an aggregation layer, and a core layer. Hosts are partitioned into VLAN subnets of a few hundred each to contain broadcasts such as ARP.
The weakness is oversubscription: the uplinks cannot carry all the traffic that the downlinks can generate. A ToR switch with 48 ports of 10 Gbit/s toward servers and four 40 Gbit/s uplinks has 480 Gbit/s down and 160 Gbit/s up, an oversubscription of 3:1. If every server in the rack sends to other racks at full speed, each gets a third of its rate. Traffic within a rack is unaffected, so applications that talk a lot should live close together. Remedies are faster uplinks (costly), placing chatty services together, and adding more parallel paths between tiers.
Leaf-spine
Section titled “Leaf-spine”Modern designs use two tiers, a Clos or leaf-spine fabric:
- Every leaf switch (ToR) connects to every spine switch.
- Hosts connect only to leaves. Spines connect only to leaves.
flowchart TB
S1["Spine 1"] --- L1["Leaf 1"]
S1 --- L2["Leaf 2"]
S1 --- L3["Leaf 3"]
S2["Spine 2"] --- L1
S2 --- L2
S2 --- L3
L1 --- R1["Rack 1 servers"]
L2 --- R2["Rack 2 servers"]
L3 --- R3["Rack 3 servers"]
Properties:
- Predictable latency. Any server reaches any server in another rack in exactly two switch hops: leaf, spine, leaf.
- Many equal paths. Between two leaves there is one path through each spine. Traffic is spread across them with ECMP (equal-cost multipath): the switch hashes the flow’s 5-tuple and picks one path, so packets of one flow stay in order while different flows use different spines.
- Easy growth. Add leaves for more racks, add spines for more bandwidth. Oversubscription is set by the ratio of leaf uplink to downlink capacity, and can be 1:1.
- Failure tolerance. Losing a spine removes one path from each pair, not connectivity.
ECMP has a catch. A hash assigns whole flows to paths, so a few large flows can land on the same link while others are idle. Operators see uneven link utilization, and per-flow hashing cannot fix it. Techniques include splitting flows, more varied hash inputs, and flowlet-level or adaptive load balancing.
Layer 3 to the rack
Section titled “Layer 3 to the rack”Large leaf-spine fabrics usually run routing, not spanning tree, between leaves and spines. Each link is a point-to-point routed link, and BGP or OSPF distributes routes. This removes blocked links, keeps broadcast domains small, and converges quickly after a failure. To let a workload keep the same layer 2 segment wherever it runs, the fabric is overlaid with VXLAN, typically with BGP EVPN as the control plane.
What changes in this environment
Section titled “What changes in this environment”Virtualization and mobility. Virtual machines move between servers and racks without changing IP address. Broadcast-based ARP over a large flat network does not scale, so data centers replace it with a directory: a service that maps a VM’s IP to the switch or hypervisor it currently sits behind, queried like DNS. Overlays do the same job with learned endpoint tables.
Centralized control. A single organization runs the whole network, so it can use SDN: simple switches forward, and a logically centralized control system computes configuration. Configuration is generated from a source of truth and pushed by automation, because manual configuration cannot keep up.
Transport. Round trips measured in microseconds and shallow switch buffers mean standard TCP behaves poorly. Losses are rare, and a timeout in a microsecond network is a disaster. Data center variants of TCP use ECN marks to react to queue buildup before any loss, and keep queues small. For storage and machine-learning traffic, RDMA over Ethernet (for example RoCE) bypasses the operating system’s network stack for lower latency, and generally needs a lossless network configured with priority flow control and ECN.
Fault domains. Cloud providers group data centers into availability zones: separate buildings, usually a few kilometers apart, with independent power and cooling and links fast enough for synchronous replication. Design applications to survive the loss of one. Your Kubernetes cluster’s zone labels map to exactly this.
Custom hardware. The largest operators build their own switches around merchant silicon and write their own control software, which is why cloud networks behave consistently but cannot be inspected like a vendor switch.