Skip to content
Documentation Background

IPsec and VPNs

A VPN makes traffic across an untrusted network look like it travels on a private one. It does this by encrypting and authenticating packets between two endpoints. IPsec does it at the network layer, so every application above it is protected without changes. It is the standard for site-to-site links between offices and cloud networks. Network hardening lists where VPNs fit. This page covers how IPsec works and how to check it.

TLS protects one connection between two applications. IPsec protects all IP traffic between two hosts or two networks. Use TLS when you control the application. Use IPsec (or WireGuard) when you need to protect whole networks, or software that has no TLS.


IPsec is a family of protocols.

ProtocolServiceNotes
ESP (encapsulating security payload)Confidentiality, integrity, source authentication, anti-replayIP protocol 50. What nearly all deployments use
AH (authentication header)Integrity, source authentication, anti-replay. No encryptionIP protocol 51. Rarely used, and incompatible with NAT
IKE (Internet Key Exchange)Negotiates algorithms and builds the keysUDP port 500, and 4500 when NAT is detected
ModeProtectsTypical use
TransportThe payload only. The original IP header staysBetween two hosts, such as a server to a database
TunnelThe whole original packet, which is wrapped in a new IP headerSite-to-site gateways and remote access VPNs

In tunnel mode the original source and destination addresses are hidden inside the encrypted payload. An eavesdropper sees only the two gateways talking. That is how two offices with private 10.x addresses can connect across the internet.


Before any packet is protected, the two ends agree on how. That agreement is a security association (SA): a one-way logical connection with its own parameters. A conversation in both directions needs two SAs.

An SA holds:

  • The protocol (ESP) and mode.
  • The encryption and integrity algorithms and their keys.
  • A 32-bit security parameter index (SPI), which labels the SA in each packet.
  • The current sequence number, for anti-replay.
  • The SA lifetime.

Two databases drive the processing:

DatabaseHoldsQuestion it answers
SPD (security policy database)Rules that match traffic by address, protocol, and portWhat should happen to this packet: protect, bypass, or discard?
SAD (security association database)The active SAsWhich keys and algorithms apply to it?

For an outgoing packet the gateway consults the SPD. If the policy says protect, it looks up the SA in the SAD and applies it. For an incoming ESP packet it finds the SA through the SPI, and then checks the result against policy.


In tunnel mode the sender builds the packet like this:

  1. Append an ESP trailer to the original IP datagram: padding to fit the cipher block size, a pad length, and a next-header field (what the payload is).
  2. Encrypt the datagram and trailer together.
  3. Prepend an ESP header: the SPI and a sequence number.
  4. Compute an authentication tag (an integrity check, in a field called the ICV) over the header and encrypted part, and append it.
  5. Put a new IP header in front, addressed gateway to gateway, with protocol 50.
| New IP header | ESP header (SPI, seq) | Original IP datagram | ESP trailer | ICV |
| (outer) | plain | encrypted, with the trailer | auth tag |

The receiver reverses these steps. It looks up the SA by SPI, checks the sequence number against a sliding window (to reject replays), verifies the tag, decrypts, removes the trailer, and forwards the inner datagram. Modern deployments use AEAD ciphers such as AES-GCM, where encryption and integrity are one operation.

The ESP sequence number is how IPsec defends against replay: the receiver keeps a window of recent numbers and drops duplicates or ones that are too old. Authentication protocols and replay explains why that is needed.


Manual keying works for two endpoints and does not scale. IKE automates it. The current version, IKEv2 (RFC 7296), has the same structure as the original two-phase design, in two exchanges:

  1. Create the IKE SA (IKE_SA_INIT). The two sides run a Diffie-Hellman exchange and agree on cryptographic algorithms. This yields a secure channel between them and a master secret. No identity has been revealed yet, and no public-key signatures are used.
  2. Authenticate and create the first child SA (IKE_AUTH). Inside the encrypted channel, each side proves its identity with a certificate and signature (or a pre-shared key) and they negotiate the IPsec SAs. Because this exchange is encrypted, a passive eavesdropper never learns who the endpoints are.
  3. Create more SAs (CREATE_CHILD_SA). New or rekeyed IPsec SAs come from the master secret using symmetric operations and, optionally, a fresh Diffie-Hellman. Creating them is cheap, because public-key math is not repeated for each.

That split exists for cost: public-key operations are expensive, so they are done once to build the IKE SA, and many IPsec SAs follow quickly. It also means keys are rotated often without re-authenticating.

Use certificates for authentication where you can (PKI and certificates). A shared pre-shared key across many sites is a weak point.


Most Linux gateways run strongSwan or Libreswan, which implement IKE and program the kernel’s IPsec stack.

Terminal window
# strongSwan: list IKE SAs and the child SAs
sudo swanctl --list-sas
# gw-to-office: #1, ESTABLISHED, IKEv2, ...
# office-net: #1, INSTALLED, TUNNEL, ESP:AES_GCM_16-256/ECP_256
# Kernel view: the SAs (keys) and policies
sudo ip xfrm state
sudo ip xfrm policy
# Bring a connection up and watch the negotiation
sudo swanctl --initiate --child office-net
sudo journalctl -u strongswan -f
# On the wire
sudo tcpdump -ni eth0 'esp or udp port 500 or udp port 4500'

A cleartext capture of a working tunnel shows ESP packets with an SPI and a sequence number, and nothing readable inside. Seeing IKE packets but no ESP means the SA formed but traffic is not matching policy or is being dropped. Seeing neither means the IKE handshake is blocked.

SymptomLikely cause
IKE never completesUDP 500 or 4500 blocked, or a mismatch in algorithms (proposals), identities, or certificates
Tunnel is up, no traffic passesThe traffic selectors (which subnets go through the tunnel) do not match on both sides, or routing and firewall rules do not send the traffic into the tunnel
Small packets work, large ones hangMTU. ESP adds tens of bytes of overhead, and the path MTU is lower. Clamp the TCP MSS on the tunnel interface (for example iptables -t mangle ... --clamp-mss-to-pmtu) or lower the MTU
Works for a while, then dropsMismatched SA lifetimes, or one side did not rekey
Fails through a home routerNAT. See below

ESP has no ports, so a NAT device cannot translate it per connection. NAT traversal solves this: IKE detects the NAT and moves to UDP port 4500, and ESP packets are wrapped in UDP. If you are debugging a remote-access VPN that works from the office and not from a hotel, check that UDP 4500 is allowed and that NAT-T is enabled.


IPsecWireGuardTLS-based VPN (OpenVPN, others)
LayerNetworkNetworkApplication (over TCP or UDP)
SetupIKE negotiation, many optionsStatic public keys, fixed algorithmsCertificates, many options
ComplexityHigh, large configuration surfaceSmall, a few hundred lines of codeModerate
InteroperabilityExcellent across vendorsGood, newerModerate
Typical useSite-to-site, clouds, routersRemote access, mesh overlaysRemote access

IPsec’s strength is that every router, firewall, and cloud provider speaks it, so it is the default for linking networks across vendors. Where both ends are Linux you control, WireGuard is simpler to run and audit. Cloud VPN gateways almost always offer IPsec.