SDN controller internals
This is an optional deep dive. SDN and OpenFlow covers the model and the basic messages. This page goes one level down: what a flow rule contains, how a switch handles many tables, and how a controller cluster stays correct when parts of it fail.
Anatomy of a flow rule
Section titled “Anatomy of a flow rule”A flow entry in an OpenFlow switch holds more than a match and an action.
| Field | Purpose |
|---|---|
| Match | Header fields to compare, with wildcards or masks |
| Priority | Breaks ties when several entries match. Highest wins |
| Instructions or actions | What to do, executed in order |
| Counters | Packets and bytes matched, and entry age |
| Idle timeout | Remove the entry if it matches nothing for this long |
| Hard timeout | Remove the entry after this long regardless |
| Cookie | A value the controller chooses to tag and find its own rules |
Timeouts keep tables from filling with rules for flows that ended. Setting a rule’s idle timeout is a design choice. Short values reclaim space but cause repeat misses and packet-in load. Long values hold stale entries.
Match fields
Section titled “Match fields”OpenFlow 1.0 matches 12 values: the ingress port plus 11 header fields.
| Layer | Fields |
|---|---|
| Ingress | Switch input port |
| Link (L2) | Source MAC, destination MAC, EtherType, VLAN ID, VLAN priority |
| Network (L3) | Source IP, destination IP, IP protocol, IP ToS |
| Transport (L4) | TCP or UDP source port, destination port |
Later versions add many more, including IPv6 fields, MPLS labels, tunnel identifiers, and metadata carried between tables, for a total of several dozen. Two fields are deliberately absent from the original list: TTL and datagram length. Each extra matchable field makes the hardware lookup more complex, and the designers traded flexibility against speed.
A wildcard field matches anything. A mask matches part of a value, such as an IP prefix. A packet that matches no entry follows the table-miss entry, which is typically “send to controller” or “drop”.
Actions
Section titled “Actions”- Output to a port, to all ports except the ingress (flood), or to the controller.
- Drop (an entry with no output action).
- Set field on any L2, L3, or L4 header field except the IP protocol, which enables NAT-style rewriting on the switch.
- Push or pop a VLAN or MPLS tag.
- Group actions, which apply an action to a set of ports (multicast, or select-one-of-many for load balancing and fast failover).
Multiple tables
Section titled “Multiple tables”Later OpenFlow versions replace the single flow table with a pipeline of tables. A packet enters table 0. A matching entry can modify the packet, write metadata, and send it on to a later table (goto-table). Each table can handle one concern:
| Table | Job |
|---|---|
| 0 | Access control and classification |
| 1 | L2 learning and forwarding |
| 2 | L3 routing |
| 3 | Load balancing and rewriting |
Splitting concerns keeps the number of entries manageable. Without tables, combining 100 access rules with 100 routes needs up to 10,000 entries, because every pair must be listed. With two tables it needs 200.
Hardware tables live in TCAM, which is fast and small, and expensive in power. A software switch like Open vSwitch uses hash-based caches in front of its full tables. The first packet of a flow goes through the slow path, which evaluates the full pipeline. The result is cached, and later packets of that flow take the fast path.
# Rules as the controller installed themsudo ovs-ofctl dump-flows br0
# Flows the fast path actually cached for trafficsudo ovs-appctl dpctl/dump-flowsWhen throughput is low but CPU on the host is high, a long run of cache misses (many short-lived flows) is a likely cause, because every miss takes the slow path.
Controller state and clustering
Section titled “Controller state and clustering”A controller keeps a model of the whole network: switches, ports, links, hosts, installed flows, counters. That model is its network information base. Applications read from it and write intent to it, and the controller turns the results into flow rules.
Running it as a single server is a single point of failure and a capacity limit, so production controllers are logically centralized but physically distributed: a cluster of servers presenting one view.
| Concern | Typical approach |
|---|---|
| Consistency | Keep the replicated state with a consensus protocol such as Raft. A write is committed when a majority of nodes accept it |
| Failover | Each switch connects to several controller nodes. One is master for that switch and the others are standby. If the master dies, another takes over |
| Scale | Add nodes to the cluster, or partition the network into domains with one controller per domain |
| Event ordering | A central store with publish and subscribe gives a single ordered log of changes, which makes failures much easier to replay and debug |
Failure domains are a design choice. One controller for the whole network is easy to reason about and has a large blast radius. Many small domains limit the damage of a bug or an outage but require coordination between them. Large operators partition by site or by fabric and keep a thin global layer on top for inter-domain routing.
What ONOS does
Section titled “What ONOS does”ONOS is an open-source controller aimed at service provider networks, and its structure follows the general design:
- Northbound: applications use an API or an intent framework. An intent says what is needed (“connect host A to host B”, “block A from B”) and ONOS works out the rules and keeps them correct as the topology changes.
- Distributed core: the network state is replicated across a cluster of servers. Adding servers adds capacity while applications still see one controller.
- Southbound: protocol-agnostic drivers let it talk to devices over OpenFlow, NETCONF, OVSDB, and others.
Intent and reconciliation
Section titled “Intent and reconciliation”Rather than telling the controller to execute steps, operators declare the state they want. A set of control loops then compares the intended state to the actual state reported by devices, and acts on the difference.
flowchart LR
I["Intended state<br/>from applications"] --> R["Reconciler"]
A["Actual state<br/>reported by devices"] --> R
R -->|"flow-mod, config"| D["Devices"]
D -->|"status, counters"| A
Large controllers split this into small services around the shared store. A topology service tracks nodes, links, and ports. A config service validates and applies device settings. A flow service reconciles the flow rules a device should have against those it reports. Applications such as routing, traffic engineering, and inter-domain routing sit above and communicate only through the store.
The benefits are practical:
- A restarted device receives its full state again without anyone re-running a script.
- Drift (a manual change on a device) is detected and corrected.
- Intent changes slowly and deliberately, while the low-level apps adjust continuously to keep reality matching it.
Kubernetes works the same way. A controller compares the desired state in the API server with what is running and acts on the difference. The pattern is general: the same idea underlies infrastructure as code and GitOps, covered in Infrastructure as code.