QoS and Traffic Shaping
Why QoS?
Section titled “Why QoS?”Networks have finite bandwidth. When the pipe is full, packets queue and then get dropped - this affects all traffic equally by default. Quality of Service (QoS) lets you prioritize some traffic over others, ensuring critical traffic (VoIP calls, video conferencing) isn’t degraded by bulk traffic (backups, downloads).
Without QoS: With QoS:┌────────────────┐ ┌────────────────┐│ VoIP (20 Kbps) │ ──mix──▶ 🚧 │ VoIP (20 Kbps) │ ──fast lane──▶ ✅│ Backup (100Mbps│ │ Backup (100Mbps) │ ──slow lane──▶ ✅│ All suffer! │ │ Both work! │└────────────────┘ └────────────────┘DSCP - Differentiated Services Code Point
Section titled “DSCP - Differentiated Services Code Point”DSCP is a 6-bit field in the IP header (within the former Type of Service byte). Routers read this field to determine how to treat the packet. The sender marks the packet, and every router along the path can honor that marking.
IPv4 Header - Type of Service (ToS) Byte:┌──────────────────────────┬─────┬───┐│ DSCP (6 bits) │ ECN │ ││ Differentiated Services │ (2b)│ │└──────────────────────────┴─────┴───┘DSCP Classes (Per-Hop Behaviors)
Section titled “DSCP Classes (Per-Hop Behaviors)”| PHB | DSCP Value | Binary | Traffic Type | Drop Probability |
|---|---|---|---|---|
| Default (BE) | 0 | 000000 | Best Effort - no priority | Standard |
| CS1 | 8 | 001000 | Background (lower than default) | High |
| AF11 | 10 | 001010 | Bulk data - low priority | Low |
| AF21 | 18 | 010010 | Bulk data - medium priority | Low |
| AF31 | 26 | 011010 | Business critical apps | Low |
| AF41 | 34 | 100010 | Video conferencing | Low |
| CS5 | 40 | 101000 | Signaling | Low |
| EF | 46 | 101110 | Expedited Forwarding - VoIP | Very Low |
| CS6 | 48 | 110000 | Network control (routing protocols) | Very Low |
| CS7 | 56 | 111000 | Network management | Very Low |
QoS Mechanisms
Section titled “QoS Mechanisms”1. Classification and Marking
Section titled “1. Classification and Marking”Before a packet can be prioritized, it must be identified:
| Method | How |
|---|---|
| DSCP/TOS marking | Read the DSCP field set by the sender |
| Port-based | TCP/UDP port number (SSH=22, SIP=5060) |
| IP address | Source or destination IP |
| Protocol | TCP, UDP, ICMP |
| Deep Packet Inspection (DPI) | Look inside the payload (detects apps using non-standard ports) |
2. Queuing
Section titled “2. Queuing”Queuing determines what to do when the link is congested:
| Algorithm | How it works | Best for |
|---|---|---|
| FIFO (First In, First Out) | No priority, pure order | Simple links, no QoS |
| WFQ (Weighted Fair Queuing) | Multiple queues, each gets bandwidth share | General QoS |
| CBWFQ (Class-Based WFQ) | Admin-defined classes each get minimum bandwidth % | Enterprise QoS |
| LLQ (Low-Latency Queuing) | Strict priority queue for real-time + CBWFQ for rest | VoIP environments |
| HTB (Hierarchical Token Bucket) | Linux’s class-based queuing | Linux servers/routers |
| HFSC | Hierarchical fair service curves | Complex multi-class scenarios |
How the schedulers differ
Section titled “How the schedulers differ”When packets wait at an output port, the scheduler picks which one goes next. Four basic disciplines cover most of the table above.
| Discipline | Rule | Risk | Linux qdisc |
|---|---|---|---|
| FIFO | Send in arrival order | One bulk flow delays everything behind it | pfifo, bfifo |
| Strict priority | Always serve the highest non-empty class first | Low classes starve if high-priority traffic never stops | prio |
| Round robin | Take one packet per class in turn | Gives equal packet counts, not equal bytes | sfq, fq (per flow) |
| Weighted fair queuing | Round robin where class i gets weight w_i | Needs sensible weights | htb, hfsc, fq_codel |
All four are work-conserving: the link never idles while a packet is queued, and an empty class is skipped at once. Priority scheduling is also non-preemptive. A packet already on the wire finishes before a higher-priority packet is sent.
With weighted fair queuing, class i is guaranteed R x w_i / sum(w_j) while all classes have traffic queued. On a 100 Mbit/s link with weights 5, 3, and 2, the classes get 50, 30, and 20 Mbit/s under full load. If the third class goes idle, its share is split among the others in proportion to their weights. In htb the guarantee is the rate and the borrowing limit is the ceil.
# Which qdisc is attached, and what is the system default?tc qdisc show dev eth0sysctl net.core.default_qdisc# net.core.default_qdisc = fq_codel3. Shaping vs Policing
Section titled “3. Shaping vs Policing”| Mechanism | What it does | Effect on excess |
|---|---|---|
| Traffic Shaping | Buffers excess traffic and sends later (smooth the burst) | Delay (no drop if buffer not full) |
| Traffic Policing | Drops or re-marks packets exceeding the rate | Drop or DSCP downgrade |
Shaping: ████░░████░░████ → ██████████████ (smoothed, delayed)
Policing: ████░░████░░████ → ████ ████ (excess dropped)Linux Traffic Control (tc)
Section titled “Linux Traffic Control (tc)”The tc command (from iproute2) implements QoS on Linux. It can:
- Set up queuing disciplines (qdiscs)
- Create traffic classes
- Add filters to classify traffic
Basic Rate Limiting with HTB
Section titled “Basic Rate Limiting with HTB”# Limit outbound traffic to 10 Mbit/s totalsudo tc qdisc add dev eth0 root handle 1: htb default 10
# Add a class with 10 Mbit/s ceilingsudo tc class add dev eth0 parent 1: classid 1:10 htb rate 10mbit
# Apply to all traffic on eth0sudo tc filter add dev eth0 protocol ip parent 1:0 u32 match ip dst 0.0.0.0/0 flowid 1:10
# View current qdisc setuptc qdisc show dev eth0tc class show dev eth0tc filter show dev eth0Priority Queuing - VoIP First
Section titled “Priority Queuing - VoIP First”# Create 3-priority prio qdiscsudo tc qdisc add dev eth0 root handle 1: prio priomap 0 0 0 0 1 1 1 1 2 2 2 2 2 2 2 2
# Band 0 (highest) = DSCP EF (VoIP)# Band 1 (medium) = DSCP AF# Band 2 (lowest) = best effort
# Filter: DSCP 46 (EF) → priority band 0sudo tc filter add dev eth0 parent 1:0 protocol ip handle 0x2e/0xfc fw flowid 1:1
# View stats (bytes/packets per queue)tc -s qdisc show dev eth0Simple Bandwidth Cap with Token Bucket Filter
Section titled “Simple Bandwidth Cap with Token Bucket Filter”# Limit a single IP to 1 Mbit/s (rate limiting for a client)sudo tc qdisc add dev eth0 root handle 1: htb default 10sudo tc class add dev eth0 parent 1: classid 1:10 htb rate 10mbit # default classsudo tc class add dev eth0 parent 1: classid 1:20 htb rate 1mbit # limited class
# Filter: traffic to 192.168.1.100 → limited classsudo tc filter add dev eth0 protocol ip parent 1:0 prio 1 \ u32 match ip dst 192.168.1.100/32 flowid 1:20
# Clean upsudo tc qdisc del dev eth0 rootMarking with iptables
Section titled “Marking with iptables”# Mark packets for QoS classification using iptables (DSCP marking)
# Mark all SSH traffic as DSCP CS6 (network control)sudo iptables -t mangle -A OUTPUT -p tcp --dport 22 -j DSCP --set-dscp-class CS6
# Mark VoIP (SIP port 5060) as EFsudo iptables -t mangle -A OUTPUT -p udp --dport 5060 -j DSCP --set-dscp 46
# Mark backup traffic as CS1 (background, below best-effort)sudo iptables -t mangle -A OUTPUT -p tcp --dport 873 -j DSCP --set-dscp-class CS1 # rsync
# View marked packetssudo iptables -t mangle -L OUTPUT -n -vTroubleshooting QoS
Section titled “Troubleshooting QoS”# Check if packets are being queued/droppedtc -s qdisc show dev eth0# Output includes: sent X bytes X pkt (dropped X, overlimits X requeues X)
# Check DSCP markings on packets leaving your machinesudo tcpdump -nn -i eth0 -e 'ip[1] & 0xfc != 0'# Shows all packets where DSCP != 0 (i.e., marked packets)
# Monitor per-class traffic rateswatch -n 1 'tc -s class show dev eth0'
# ss - check for large receive queues (signs of congestion)ss -tnp | awk 'NR>1 && $2+0 > 0 {print "Recv-Q:", $2, $5}'Bufferbloat
Section titled “Bufferbloat”Bufferbloat happens when routers/modems have excessively large buffers. Traffic doesn’t get dropped fast enough, so it sits in the buffer causing high latency:
No buffer bloat: 20ms RTTWith bufferbloat: 500ms RTT (buffer filling = 480ms of queued data)Fix: CoDel (Controlled Delay) or FQ-CoDel manages queues intelligently:
# Replace default qdisc with FQ-CoDel (reduced bufferbloat)sudo tc qdisc replace dev eth0 root fq_codel
# Or cake (even better, does fairness + bufferbloat management together)sudo tc qdisc replace dev eth0 root cake bandwidth 100mbitWhy buffers fill and stay full
Section titled “Why buffers fill and stay full”A buffer absorbs bursts, which lowers loss. It also adds delay. Ten times the buffering can mean ten times the queueing delay, and a longer RTT makes TCP senders slower to notice congestion.
The standing queue comes from ACK clocking. A burst fills the queue. Each returning ACK lets the sender release one more packet, so the queue stays full for as long as the transfer runs. Latency-sensitive traffic such as calls and games shares that queue and waits behind the bulk transfer.
Sizing buffers
Section titled “Sizing buffers”The classic rule sets the buffer to the bandwidth-delay product, B = RTT x C. A 10 Gbit/s link with a 250 ms RTT needs 2.5 Gbit. When many independent TCP flows share a backbone link, the aggregate smooths out and the need drops to B = RTT x C / sqrt(N). With 10,000 flows, that 2.5 Gbit falls to 25 Mbit. Edge devices have few flows, so they gain little from the second rule and are the usual place for bufferbloat. Router switching and queueing shows where both rules come from.
Active queue management
Section titled “Active queue management”A drop-tail queue drops only when it is completely full, so senders learn about congestion late. Active queue management (AQM) drops or marks packets earlier, while the queue is still short.
| Algorithm | Signal it reacts to | Notes |
|---|---|---|
| RED | Average queue length. Drops with rising probability | The original AQM. Hard to tune, so rarely used on its own |
| CoDel | Time packets spend in the queue (target about 5 ms) | No tuning parameters. The basis of fq_codel |
| PIE | Estimated queueing delay | Used by DOCSIS 3.1 cable modems |
AQM can mark packets with ECN instead of dropping them, so the sender slows down without losing data. See TCP congestion control for the sender side.
Measuring bufferbloat
Section titled “Measuring bufferbloat”Compare latency on an idle link with latency under load.
# Terminal 1: saturate the linkiperf3 -c iperf.example.net -t 30
# Terminal 2: watch latency while it runsping -i 0.2 -c 100 198.51.100.1# Idle: ~20 ms. Under load with bufferbloat: 300 to 500 ms.
# Drops and ECN marks from the qdisctc -s qdisc show dev eth0