Skip to content
Documentation Background

HTTP and web protocols

HTTP is the application-layer protocol behind browsers, REST APIs, and most service-to-service calls. A client sends a request, the server sends one response, and neither side remembers anything about earlier requests. Sessions, caching, and connection reuse are all layered on top of that stateless exchange.

A web page is a set of objects: a base HTML file plus the images, scripts, and stylesheets it references. Each object has a URL made of a hostname (which server) and a path (which object on it). The client fetches every object with its own request.


HTTP/1.1 messages are plain text. A message is a start line, header lines, a blank line, and an optional body. curl -v prints both directions, with > for the request and < for the response.

Terminal window
curl -v http://app.example.com/index.html
# > GET /index.html HTTP/1.1
# > Host: app.example.com
# > User-Agent: curl/8.5.0
# > Accept: */*
# >
# < HTTP/1.1 200 OK
# < Content-Type: text/html; charset=utf-8
# < Content-Length: 1256
# < Cache-Control: max-age=3600
# < ETag: "5d8c72a5"

The request line holds the method, the path, and the protocol version. The status line of the response holds the version, a numeric status code, and a short phrase.

MethodPurposeSafeIdempotent
GETFetch an object. No body.YesYes
HEADSame as GET but returns headers only. Useful for checking size, type, or freshnessYesYes
POSTSubmit data for processing, such as a form or a new recordNoNo
PUTCreate or replace the object at the URLNoYes
PATCHApply a partial changeNoNo
DELETERemove the objectNoYes

Idempotent means repeating the request leaves the same end state, which is why clients and proxies can retry GET, PUT, and DELETE after a timeout but must be careful with POST.

RangeMeaningCommon codes
2xxSuccess200 OK, 201 Created, 204 No Content
3xxRedirect or not modified301 Moved Permanently, 302 Found, 304 Not Modified
4xxClient error400 Bad Request, 401 Unauthorized, 403 Forbidden, 404 Not Found, 429 Too Many Requests
5xxServer error500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable, 504 Gateway Timeout

Behind a reverse proxy or load balancer, the 5xx codes tell you where to look. 502 means the proxy reached the upstream but got an invalid response or a refused connection. 504 means the upstream did not answer within the proxy’s timeout.

HeaderDirectionPurpose
HostRequestWhich site the request is for. Lets one IP serve many sites
AuthorizationRequestCredentials such as a bearer token
Content-TypeBothMedia type of the body, such as application/json
Content-LengthBothBody size in bytes
LocationResponseTarget of a redirect
Set-Cookie / CookieResponse / RequestSession state (see below)
Cache-Control, ETagResponseCaching rules and a version tag for the object

Every new TCP connection costs a handshake round trip before any HTTP data moves, plus more round trips if TLS is on top. How many requests share a connection therefore decides much of the load time.

  • Non-persistent (HTTP/1.0) - one request per connection. The server closes the connection after each response.
  • Persistent (HTTP/1.1 default) - the connection stays open after a response and later requests reuse it. The server closes it after an idle timeout, or when either side sends Connection: close.

Take a page with one HTML file and 10 images on a 50 ms round-trip path, counting only network round trips. With a new connection per object, each object costs 2 round trips (handshake plus request), so 11 objects cost 22 round trips, or 1.1 seconds. With one persistent connection it costs 1 round trip for the handshake and 1 per request, so 12 round trips, or 600 ms.

Persistent connections still deliver responses in order. A large response at the front of the connection holds back the small ones behind it. This is head-of-line blocking at the application level. HTTP/1.1 clients work around it by opening several connections to the same host in parallel (browsers typically use six). Pipelining requests without waiting for responses exists in the spec but is turned off in practice for the same reason.


Because HTTP is stateless, a server needs another way to recognize a returning user. Cookies add that:

  1. The server responds with a Set-Cookie header carrying an identifier.
  2. The browser stores it and sends it back in a Cookie header on every later request to that site.
  3. The server uses the identifier to find the user’s session data, usually in a database or a shared cache.
HTTP/1.1 200 OK
Set-Cookie: sid=9f2c41ab; Max-Age=3600; Secure; HttpOnly; SameSite=Lax
AttributeEffect
SecureSent only over HTTPS
HttpOnlyNot readable by JavaScript, which limits what an injected script can steal
SameSiteControls whether the cookie is sent on cross-site requests. Lax blocks most cross-site POSTs, Strict blocks all
Max-Age / ExpiresLifetime. Without either, the cookie lasts until the browser closes
Domain / PathWhich hosts and paths receive the cookie

Statelessness is also what lets a load balancer send consecutive requests to different servers. If session data lives in a server’s own memory, a user who lands on another server looks logged out. Keep sessions in a shared store, or use sticky sessions as a stopgap. See The twelve-factor app for the design reasoning.


A cache keeps a copy of a response and serves it for later requests, which removes the server round trip entirely. Caches exist in the browser, in reverse proxies, and in CDNs (see CDNs and video streaming).

The server sets the rules with Cache-Control on the response:

DirectiveMeaning
max-age=3600The copy is fresh for 3600 seconds. No request is needed until then
no-cacheA copy may be stored, but must be revalidated with the server before each use
no-storeDo not store the response at all. Use for sensitive data
private / publicOnly the end user’s browser may cache it, or any shared cache may

When a copy expires, the cache does not have to download the object again. It sends a conditional request carrying the version it holds. If the object has not changed, the server answers 304 Not Modified with no body, and the cache reuses its copy.

Terminal window
# See the validators the server sends
curl -sI https://app.example.com/logo.png | grep -iE 'etag|cache-control|last-modified'
# etag: "5d8c72a5"
# cache-control: max-age=3600
# Revalidate with the tag (use the value from the previous response)
curl -sI -H 'If-None-Match: "5d8c72a5"' https://app.example.com/logo.png | head -1
# HTTP/1.1 304 Not Modified

ETag with If-None-Match is the stronger validator. Last-Modified with If-Modified-Since works on timestamps and is the older equivalent.


HTTP/2 and HTTP/3 keep the same methods, status codes, and headers. They change how messages are encoded and carried, to fix the head-of-line blocking and connection costs above.

HTTP/1.1HTTP/2HTTP/3
StandardRFC 9112RFC 9113RFC 9114
TransportTCPTCPQUIC over UDP
EncodingTextBinary framesBinary frames
Requests per connectionOne at a time, in orderMany in parallel (streams)Many in parallel (streams)
Application-level blockingYesNoNo
Transport-level blockingMitigated by parallel connectionsYes, one lost TCP segment stalls all streamsNo, loss only stalls the affected stream
Round trips before first request (new connection)2 (TCP + TLS 1.3), 3 with TLS 1.22 (TCP + TLS 1.3)1 (0 for a returning client)

HTTP/2 splits each message into small binary frames and interleaves frames from many responses over a single TCP connection. A small stylesheet no longer waits behind a large video. It also compresses headers, which matters because the same cookies and user agent repeat on every request. Server push, which let a server send objects before they were requested, was dropped by major browsers and should not be relied on.

One weakness remains. All streams share one TCP connection, and TCP delivers bytes in order. A single lost segment holds up every stream until it is retransmitted.

HTTP/3 runs on QUIC, a transport built on UDP. QUIC handles loss per stream, so a lost packet delays only the stream it belongs to. Several other properties come with it:

  • TLS 1.3 built in - connection setup and key exchange happen in one handshake, so a new connection needs one round trip, and a returning client can send data in the first flight (0-RTT).
  • Connection IDs - a connection is identified by an ID, not by the IP and port tuple. A phone that moves from Wi-Fi to cellular keeps its session.
  • Congestion control in user space - the algorithm is part of the QUIC library, so it can change without an operating system update. See TCP congestion control.

A server advertises HTTP/3 with an Alt-Svc header. Browsers try QUIC and fall back to TCP silently if UDP port 443 is blocked, so a blocked UDP path does not show up as an outage.

Terminal window
# Which version was negotiated?
curl -s -o /dev/null -w '%{http_version}\n' https://app.example.com/
# 2
# Force a version (HTTP/3 needs a curl built with QUIC support)
curl -sI --http1.1 https://app.example.com/ | head -1
curl -sI --http2 https://app.example.com/ | head -1
curl -sI --http3 https://app.example.com/ | head -1
# Does the server advertise HTTP/3?
curl -sI https://app.example.com/ | grep -i alt-svc
# alt-svc: h3=":443"; ma=86400

curl can break a request into its phases, which shows where the time goes.

Terminal window
curl -s -o /dev/null https://app.example.com/ \
-w 'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n'
# dns=0.012 connect=0.041 tls=0.093 ttfb=0.162 total=0.170

Each value is cumulative. A large gap between connect and tls points to the TLS handshake, and a large gap before ttfb means the server was slow to respond. See TLS and HTTPS for the handshake itself.


SymptomLikely causeFix
502 Bad Gateway from the proxyUpstream is down, refused the connection, or returned an invalid responseCheck the upstream service and port, then read the proxy error log
504 Gateway TimeoutUpstream took longer than the proxy’s timeoutFind the slow request with curl -w timings, or raise the proxy timeout if the delay is expected
Redirect loopApp and proxy both redirect HTTP to HTTPS, or the app cannot see the original schemeTrace with curl -IL. Pass X-Forwarded-Proto from the proxy and trust it in the app
Users still see old assets after a deployLong max-age on a file whose name did not changeHash filenames, or purge the CDN and shorten max-age
Users are logged out at random behind a load balancerSessions held in one server’s memoryUse a shared session store
New connections feel slow, repeat requests are fastHandshake cost, with connections not reusedEnable keep-alive, use HTTP/2, and pool connections in API clients
HTTP/3 never gets usedUDP 443 blocked by a firewallAllow UDP 443. Clients fall back to TCP, so nothing breaks