IPsec and VPNs
A VPN makes traffic across an untrusted network look like it travels on a private one. It does this by encrypting and authenticating packets between two endpoints. IPsec does it at the network layer, so every application above it is protected without changes. It is the standard for site-to-site links between offices and cloud networks. Network hardening lists where VPNs fit. This page covers how IPsec works and how to check it.
TLS protects one connection between two applications. IPsec protects all IP traffic between two hosts or two networks. Use TLS when you control the application. Use IPsec (or WireGuard) when you need to protect whole networks, or software that has no TLS.
The pieces
Section titled “The pieces”IPsec is a family of protocols.
| Protocol | Service | Notes |
|---|---|---|
| ESP (encapsulating security payload) | Confidentiality, integrity, source authentication, anti-replay | IP protocol 50. What nearly all deployments use |
| AH (authentication header) | Integrity, source authentication, anti-replay. No encryption | IP protocol 51. Rarely used, and incompatible with NAT |
| IKE (Internet Key Exchange) | Negotiates algorithms and builds the keys | UDP port 500, and 4500 when NAT is detected |
Transport and tunnel mode
Section titled “Transport and tunnel mode”| Mode | Protects | Typical use |
|---|---|---|
| Transport | The payload only. The original IP header stays | Between two hosts, such as a server to a database |
| Tunnel | The whole original packet, which is wrapped in a new IP header | Site-to-site gateways and remote access VPNs |
In tunnel mode the original source and destination addresses are hidden inside the encrypted payload. An eavesdropper sees only the two gateways talking. That is how two offices with private 10.x addresses can connect across the internet.
Security associations
Section titled “Security associations”Before any packet is protected, the two ends agree on how. That agreement is a security association (SA): a one-way logical connection with its own parameters. A conversation in both directions needs two SAs.
An SA holds:
- The protocol (ESP) and mode.
- The encryption and integrity algorithms and their keys.
- A 32-bit security parameter index (SPI), which labels the SA in each packet.
- The current sequence number, for anti-replay.
- The SA lifetime.
Two databases drive the processing:
| Database | Holds | Question it answers |
|---|---|---|
| SPD (security policy database) | Rules that match traffic by address, protocol, and port | What should happen to this packet: protect, bypass, or discard? |
| SAD (security association database) | The active SAs | Which keys and algorithms apply to it? |
For an outgoing packet the gateway consults the SPD. If the policy says protect, it looks up the SA in the SAD and applies it. For an incoming ESP packet it finds the SA through the SPI, and then checks the result against policy.
The ESP packet
Section titled “The ESP packet”In tunnel mode the sender builds the packet like this:
- Append an ESP trailer to the original IP datagram: padding to fit the cipher block size, a pad length, and a next-header field (what the payload is).
- Encrypt the datagram and trailer together.
- Prepend an ESP header: the SPI and a sequence number.
- Compute an authentication tag (an integrity check, in a field called the ICV) over the header and encrypted part, and append it.
- Put a new IP header in front, addressed gateway to gateway, with protocol 50.
| New IP header | ESP header (SPI, seq) | Original IP datagram | ESP trailer | ICV || (outer) | plain | encrypted, with the trailer | auth tag |The receiver reverses these steps. It looks up the SA by SPI, checks the sequence number against a sliding window (to reject replays), verifies the tag, decrypts, removes the trailer, and forwards the inner datagram. Modern deployments use AEAD ciphers such as AES-GCM, where encryption and integrity are one operation.
The ESP sequence number is how IPsec defends against replay: the receiver keeps a window of recent numbers and drops duplicates or ones that are too old. Authentication protocols and replay explains why that is needed.
IKE: building the SAs
Section titled “IKE: building the SAs”Manual keying works for two endpoints and does not scale. IKE automates it. The current version, IKEv2 (RFC 7296), has the same structure as the original two-phase design, in two exchanges:
- Create the IKE SA (
IKE_SA_INIT). The two sides run a Diffie-Hellman exchange and agree on cryptographic algorithms. This yields a secure channel between them and a master secret. No identity has been revealed yet, and no public-key signatures are used. - Authenticate and create the first child SA (
IKE_AUTH). Inside the encrypted channel, each side proves its identity with a certificate and signature (or a pre-shared key) and they negotiate the IPsec SAs. Because this exchange is encrypted, a passive eavesdropper never learns who the endpoints are. - Create more SAs (
CREATE_CHILD_SA). New or rekeyed IPsec SAs come from the master secret using symmetric operations and, optionally, a fresh Diffie-Hellman. Creating them is cheap, because public-key math is not repeated for each.
That split exists for cost: public-key operations are expensive, so they are done once to build the IKE SA, and many IPsec SAs follow quickly. It also means keys are rotated often without re-authenticating.
Use certificates for authentication where you can (PKI and certificates). A shared pre-shared key across many sites is a weak point.
Operating IPsec on Linux
Section titled “Operating IPsec on Linux”Most Linux gateways run strongSwan or Libreswan, which implement IKE and program the kernel’s IPsec stack.
# strongSwan: list IKE SAs and the child SAssudo swanctl --list-sas# gw-to-office: #1, ESTABLISHED, IKEv2, ...# office-net: #1, INSTALLED, TUNNEL, ESP:AES_GCM_16-256/ECP_256
# Kernel view: the SAs (keys) and policiessudo ip xfrm statesudo ip xfrm policy
# Bring a connection up and watch the negotiationsudo swanctl --initiate --child office-netsudo journalctl -u strongswan -f
# On the wiresudo tcpdump -ni eth0 'esp or udp port 500 or udp port 4500'A cleartext capture of a working tunnel shows ESP packets with an SPI and a sequence number, and nothing readable inside. Seeing IKE packets but no ESP means the SA formed but traffic is not matching policy or is being dropped. Seeing neither means the IKE handshake is blocked.
Common problems
Section titled “Common problems”| Symptom | Likely cause |
|---|---|
| IKE never completes | UDP 500 or 4500 blocked, or a mismatch in algorithms (proposals), identities, or certificates |
| Tunnel is up, no traffic passes | The traffic selectors (which subnets go through the tunnel) do not match on both sides, or routing and firewall rules do not send the traffic into the tunnel |
| Small packets work, large ones hang | MTU. ESP adds tens of bytes of overhead, and the path MTU is lower. Clamp the TCP MSS on the tunnel interface (for example iptables -t mangle ... --clamp-mss-to-pmtu) or lower the MTU |
| Works for a while, then drops | Mismatched SA lifetimes, or one side did not rekey |
| Fails through a home router | NAT. See below |
ESP has no ports, so a NAT device cannot translate it per connection. NAT traversal solves this: IKE detects the NAT and moves to UDP port 4500, and ESP packets are wrapped in UDP. If you are debugging a remote-access VPN that works from the office and not from a hotel, check that UDP 4500 is allowed and that NAT-T is enabled.
IPsec and its alternatives
Section titled “IPsec and its alternatives”| IPsec | WireGuard | TLS-based VPN (OpenVPN, others) | |
|---|---|---|---|
| Layer | Network | Network | Application (over TCP or UDP) |
| Setup | IKE negotiation, many options | Static public keys, fixed algorithms | Certificates, many options |
| Complexity | High, large configuration surface | Small, a few hundred lines of code | Moderate |
| Interoperability | Excellent across vendors | Good, newer | Moderate |
| Typical use | Site-to-site, clouds, routers | Remote access, mesh overlays | Remote access |
IPsec’s strength is that every router, firewall, and cloud provider speaks it, so it is the default for linking networks across vendors. Where both ends are Linux you control, WireGuard is simpler to run and audit. Cloud VPN gateways almost always offer IPsec.