HTTP and web protocols
HTTP is the application-layer protocol behind browsers, REST APIs, and most service-to-service calls. A client sends a request, the server sends one response, and neither side remembers anything about earlier requests. Sessions, caching, and connection reuse are all layered on top of that stateless exchange.
A web page is a set of objects: a base HTML file plus the images, scripts, and stylesheets it references. Each object has a URL made of a hostname (which server) and a path (which object on it). The client fetches every object with its own request.
Request and response format
Section titled “Request and response format”HTTP/1.1 messages are plain text. A message is a start line, header lines, a blank line, and an optional body. curl -v prints both directions, with > for the request and < for the response.
curl -v http://app.example.com/index.html
# > GET /index.html HTTP/1.1# > Host: app.example.com# > User-Agent: curl/8.5.0# > Accept: */*# ># < HTTP/1.1 200 OK# < Content-Type: text/html; charset=utf-8# < Content-Length: 1256# < Cache-Control: max-age=3600# < ETag: "5d8c72a5"The request line holds the method, the path, and the protocol version. The status line of the response holds the version, a numeric status code, and a short phrase.
Methods
Section titled “Methods”| Method | Purpose | Safe | Idempotent |
|---|---|---|---|
| GET | Fetch an object. No body. | Yes | Yes |
| HEAD | Same as GET but returns headers only. Useful for checking size, type, or freshness | Yes | Yes |
| POST | Submit data for processing, such as a form or a new record | No | No |
| PUT | Create or replace the object at the URL | No | Yes |
| PATCH | Apply a partial change | No | No |
| DELETE | Remove the object | No | Yes |
Idempotent means repeating the request leaves the same end state, which is why clients and proxies can retry GET, PUT, and DELETE after a timeout but must be careful with POST.
Status codes
Section titled “Status codes”| Range | Meaning | Common codes |
|---|---|---|
| 2xx | Success | 200 OK, 201 Created, 204 No Content |
| 3xx | Redirect or not modified | 301 Moved Permanently, 302 Found, 304 Not Modified |
| 4xx | Client error | 400 Bad Request, 401 Unauthorized, 403 Forbidden, 404 Not Found, 429 Too Many Requests |
| 5xx | Server error | 500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable, 504 Gateway Timeout |
Behind a reverse proxy or load balancer, the 5xx codes tell you where to look. 502 means the proxy reached the upstream but got an invalid response or a refused connection. 504 means the upstream did not answer within the proxy’s timeout.
Headers worth knowing
Section titled “Headers worth knowing”| Header | Direction | Purpose |
|---|---|---|
Host | Request | Which site the request is for. Lets one IP serve many sites |
Authorization | Request | Credentials such as a bearer token |
Content-Type | Both | Media type of the body, such as application/json |
Content-Length | Both | Body size in bytes |
Location | Response | Target of a redirect |
Set-Cookie / Cookie | Response / Request | Session state (see below) |
Cache-Control, ETag | Response | Caching rules and a version tag for the object |
Connection reuse
Section titled “Connection reuse”Every new TCP connection costs a handshake round trip before any HTTP data moves, plus more round trips if TLS is on top. How many requests share a connection therefore decides much of the load time.
- Non-persistent (HTTP/1.0) - one request per connection. The server closes the connection after each response.
- Persistent (HTTP/1.1 default) - the connection stays open after a response and later requests reuse it. The server closes it after an idle timeout, or when either side sends
Connection: close.
Take a page with one HTML file and 10 images on a 50 ms round-trip path, counting only network round trips. With a new connection per object, each object costs 2 round trips (handshake plus request), so 11 objects cost 22 round trips, or 1.1 seconds. With one persistent connection it costs 1 round trip for the handshake and 1 per request, so 12 round trips, or 600 ms.
Persistent connections still deliver responses in order. A large response at the front of the connection holds back the small ones behind it. This is head-of-line blocking at the application level. HTTP/1.1 clients work around it by opening several connections to the same host in parallel (browsers typically use six). Pipelining requests without waiting for responses exists in the spec but is turned off in practice for the same reason.
State with cookies
Section titled “State with cookies”Because HTTP is stateless, a server needs another way to recognize a returning user. Cookies add that:
- The server responds with a
Set-Cookieheader carrying an identifier. - The browser stores it and sends it back in a
Cookieheader on every later request to that site. - The server uses the identifier to find the user’s session data, usually in a database or a shared cache.
HTTP/1.1 200 OKSet-Cookie: sid=9f2c41ab; Max-Age=3600; Secure; HttpOnly; SameSite=Lax| Attribute | Effect |
|---|---|
Secure | Sent only over HTTPS |
HttpOnly | Not readable by JavaScript, which limits what an injected script can steal |
SameSite | Controls whether the cookie is sent on cross-site requests. Lax blocks most cross-site POSTs, Strict blocks all |
Max-Age / Expires | Lifetime. Without either, the cookie lasts until the browser closes |
Domain / Path | Which hosts and paths receive the cookie |
Statelessness is also what lets a load balancer send consecutive requests to different servers. If session data lives in a server’s own memory, a user who lands on another server looks logged out. Keep sessions in a shared store, or use sticky sessions as a stopgap. See The twelve-factor app for the design reasoning.
Caching
Section titled “Caching”A cache keeps a copy of a response and serves it for later requests, which removes the server round trip entirely. Caches exist in the browser, in reverse proxies, and in CDNs (see CDNs and video streaming).
The server sets the rules with Cache-Control on the response:
| Directive | Meaning |
|---|---|
max-age=3600 | The copy is fresh for 3600 seconds. No request is needed until then |
no-cache | A copy may be stored, but must be revalidated with the server before each use |
no-store | Do not store the response at all. Use for sensitive data |
private / public | Only the end user’s browser may cache it, or any shared cache may |
When a copy expires, the cache does not have to download the object again. It sends a conditional request carrying the version it holds. If the object has not changed, the server answers 304 Not Modified with no body, and the cache reuses its copy.
# See the validators the server sendscurl -sI https://app.example.com/logo.png | grep -iE 'etag|cache-control|last-modified'# etag: "5d8c72a5"# cache-control: max-age=3600
# Revalidate with the tag (use the value from the previous response)curl -sI -H 'If-None-Match: "5d8c72a5"' https://app.example.com/logo.png | head -1# HTTP/1.1 304 Not ModifiedETag with If-None-Match is the stronger validator. Last-Modified with If-Modified-Since works on timestamps and is the older equivalent.
HTTP versions
Section titled “HTTP versions”HTTP/2 and HTTP/3 keep the same methods, status codes, and headers. They change how messages are encoded and carried, to fix the head-of-line blocking and connection costs above.
| HTTP/1.1 | HTTP/2 | HTTP/3 | |
|---|---|---|---|
| Standard | RFC 9112 | RFC 9113 | RFC 9114 |
| Transport | TCP | TCP | QUIC over UDP |
| Encoding | Text | Binary frames | Binary frames |
| Requests per connection | One at a time, in order | Many in parallel (streams) | Many in parallel (streams) |
| Application-level blocking | Yes | No | No |
| Transport-level blocking | Mitigated by parallel connections | Yes, one lost TCP segment stalls all streams | No, loss only stalls the affected stream |
| Round trips before first request (new connection) | 2 (TCP + TLS 1.3), 3 with TLS 1.2 | 2 (TCP + TLS 1.3) | 1 (0 for a returning client) |
HTTP/2
Section titled “HTTP/2”HTTP/2 splits each message into small binary frames and interleaves frames from many responses over a single TCP connection. A small stylesheet no longer waits behind a large video. It also compresses headers, which matters because the same cookies and user agent repeat on every request. Server push, which let a server send objects before they were requested, was dropped by major browsers and should not be relied on.
One weakness remains. All streams share one TCP connection, and TCP delivers bytes in order. A single lost segment holds up every stream until it is retransmitted.
HTTP/3 and QUIC
Section titled “HTTP/3 and QUIC”HTTP/3 runs on QUIC, a transport built on UDP. QUIC handles loss per stream, so a lost packet delays only the stream it belongs to. Several other properties come with it:
- TLS 1.3 built in - connection setup and key exchange happen in one handshake, so a new connection needs one round trip, and a returning client can send data in the first flight (0-RTT).
- Connection IDs - a connection is identified by an ID, not by the IP and port tuple. A phone that moves from Wi-Fi to cellular keeps its session.
- Congestion control in user space - the algorithm is part of the QUIC library, so it can change without an operating system update. See TCP congestion control.
A server advertises HTTP/3 with an Alt-Svc header. Browsers try QUIC and fall back to TCP silently if UDP port 443 is blocked, so a blocked UDP path does not show up as an outage.
# Which version was negotiated?curl -s -o /dev/null -w '%{http_version}\n' https://app.example.com/# 2
# Force a version (HTTP/3 needs a curl built with QUIC support)curl -sI --http1.1 https://app.example.com/ | head -1curl -sI --http2 https://app.example.com/ | head -1curl -sI --http3 https://app.example.com/ | head -1
# Does the server advertise HTTP/3?curl -sI https://app.example.com/ | grep -i alt-svc# alt-svc: h3=":443"; ma=86400Measuring a request
Section titled “Measuring a request”curl can break a request into its phases, which shows where the time goes.
curl -s -o /dev/null https://app.example.com/ \ -w 'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n'
# dns=0.012 connect=0.041 tls=0.093 ttfb=0.162 total=0.170Each value is cumulative. A large gap between connect and tls points to the TLS handshake, and a large gap before ttfb means the server was slow to respond. See TLS and HTTPS for the handshake itself.
Troubleshooting
Section titled “Troubleshooting”| Symptom | Likely cause | Fix |
|---|---|---|
502 Bad Gateway from the proxy | Upstream is down, refused the connection, or returned an invalid response | Check the upstream service and port, then read the proxy error log |
504 Gateway Timeout | Upstream took longer than the proxy’s timeout | Find the slow request with curl -w timings, or raise the proxy timeout if the delay is expected |
| Redirect loop | App and proxy both redirect HTTP to HTTPS, or the app cannot see the original scheme | Trace with curl -IL. Pass X-Forwarded-Proto from the proxy and trust it in the app |
| Users still see old assets after a deploy | Long max-age on a file whose name did not change | Hash filenames, or purge the CDN and shorten max-age |
| Users are logged out at random behind a load balancer | Sessions held in one server’s memory | Use a shared session store |
| New connections feel slow, repeat requests are fast | Handshake cost, with connections not reused | Enable keep-alive, use HTTP/2, and pool connections in API clients |
| HTTP/3 never gets used | UDP 443 blocked by a firewall | Allow UDP 443. Clients fall back to TCP, so nothing breaks |