The network path of one request

DNS: what actually resolves, and where does it go wrong?

The resolution order is: the browser's own cache → the OS cache (and /etc/hosts) → the configured resolver (your ISP, 8.8.8.8, or in a container, the cluster DNS service) → which recursively queries a root server, a TLD server, then the authoritative nameserver.

What matters operationally:

• TTL is a promise you cannot revoke. Lower the TTL before a planned migration, not during. Many resolvers and some client libraries ignore short TTLs anyway.
• Java and some runtimes cache DNS forever by default (networkaddress.cache.ttl), which breaks failover to a new IP entirely. Node caches nothing at the process level, so a hot service can generate enormous DNS query volume.
• In Kubernetes, ndots: 5 in the default resolv.conf means any name with fewer than five dots is tried against every search domain first — so api.example.com may generate four failing queries before the correct one. This is a classic source of mysterious tail latency, fixed by a trailing dot (api.example.com.) or a custom dnsConfig.
• DNS is usually UDP; responses over 512 bytes need EDNS0 or fall back to TCP, and a firewall blocking DNS-over-TCP produces failures that only appear for large responses.

Cost: 0-100 ms, and zero when cached. dns-prefetch and preconnect exist to move it off the critical path.

TCP: the handshake, congestion control, and why the first request is slow

client ─ SYN ──────────────→ server        1 RTT to establish; nothing useful is sent yet
client ←─── SYN-ACK ───────── server
client ─ ACK + data ───────→ server

Then the parts people skip:

Slow start. TCP does not know the available bandwidth, so it begins with a small congestion window — typically 10 packets, about 14 KB — and doubles it each round trip until loss occurs. So a 100 KB response over a 100 ms RTT link takes several round trips regardless of bandwidth. This is why the first bytes matter more than total bandwidth, why keeping responses under ~14 KB gets them in the first round trip, and why bandwidth upgrades do so little for page load while latency reductions do a lot.

Congestion control algorithm. Traditional CUBIC treats packet loss as the congestion signal, which behaves badly on lossy wireless links (loss ≠ congestion) and on deep buffers. BBR models bottleneck bandwidth and round-trip time instead, and switching a server to BBR is a real, measurable improvement for users on mobile networks. Worth naming — it signals you have tuned something.

Head-of-line blocking at the TCP layer. TCP guarantees ordered delivery, so a single lost packet stalls everything behind it in that connection until it is retransmitted — even data belonging to unrelated HTTP/2 streams. That is the limitation HTTP/3 exists to fix.

Nagle's algorithm buffers small writes to avoid tiny packets, and interacts badly with delayed ACKs to add up to 40 ms of latency on request-response protocols. TCP_NODELAY disables it, and most modern servers and clients already set it.

TLS: what the handshake costs and what it establishes

TLS 1.3 (the current standard) completes in one round trip: the client sends its key share optimistically with the ClientHello, the server replies with its own share, its certificate and Finished, and application data can flow. TLS 1.2 needed two round trips.

What happens inside, briefly, because interviewers do probe it: key exchange (ECDHE, giving forward secrecy — a compromised server key cannot decrypt recorded past sessions), certificate verification against the trust store including hostname and validity checks and often OCSP stapling, and ALPN, which is how the client and server agree on HTTP/2 versus HTTP/1.1 during the handshake rather than paying an extra negotiation.

• Session resumption with a pre-shared key skips the key exchange, and 0-RTT lets the client send data with the very first packet. 0-RTT data is replayable by an attacker, so it must only be used for idempotent requests — a genuinely important detail that most candidates do not know.
• SNI carries the hostname in the clear (unless ECH is in use), which is how one IP serves many certificates and also how networks censor by hostname.
• Terminating TLS at a load balancer means the connection behind it is a different connection: X-Forwarded-For/Proto become the only way to know the real client, and trusting those headers from an untrusted source is a spoofing vulnerability.

Cost: 1 RTT on top of TCP's 1 RTT, so a cold HTTPS connection to a server 100 ms away costs ~200 ms before a single byte of your request is processed. That is the number that justifies connection reuse.

HTTP/1.1 vs /2 vs /3, precisely

HTTP/1.1: one request in flight per connection. Pipelining was specified and never worked in practice. Browsers therefore open ~6 connections per origin, and each pays its own TCP+TLS handshake and its own slow start. Hence the old workarounds: sprites, concatenation, domain sharding, inlining.

HTTP/2: one connection, many streams, multiplexed as interleaved binary frames. Header compression (HPACK) removes the repeated kilobytes of cookies and user-agent on every request. Server push existed and has been removed from browsers — it was almost always a net loss because the server could not know what the client had cached; 103 Early Hints is the replacement worth knowing.

Because HTTP/2 removes application-level head-of-line blocking, the old optimisations invert: many small cached files can now beat one big bundle, and domain sharding actively hurts by forcing extra connections. But TCP-level head-of-line blocking remains — one lost packet stalls every stream on that connection, which is why HTTP/2 can be worse than HTTP/1.1 on a lossy network.

HTTP/3: the same semantics over QUIC, which runs on UDP and implements streams, loss recovery and congestion control itself. Consequences: independent streams, so a lost packet affects only its own stream; the transport and TLS handshakes are combined into 1 RTT (0 with resumption); and connection migration by connection ID, so switching from Wi-Fi to cellular does not break the connection. Costs: UDP is blocked or deprioritised on some networks (hence the fallback), and per-packet CPU cost is higher because the stack is in userspace.

Why does connection pooling matter so much?

Because handshakes are per-connection and expensive, and pool exhaustion is one of the most common backend failure modes.

# Every call: DNS + TCP + TLS. Adds ~200 ms and leaks file descriptors under load.
requests.get(url)

# One pooled, keep-alive session for the process lifetime.
session = requests.Session()          # or httpx.AsyncClient / a single fetch agent

The mechanics to be able to discuss:

• Keep-alive reuses an established connection for subsequent requests, skipping both handshakes and the slow-start ramp. The idle timeout must be shorter on the client than on the server, or the client will occasionally send a request into a connection the server has already closed — producing intermittent, unreproducible connection resets. This is a real and frequently misdiagnosed bug.
• Pool sizing is concurrency × instances, and it must be reconciled against the downstream limits — a database's max_connections, a partner API's per-client cap. Twenty pods with a pool of 20 is 400 connections, whether or not anyone planned it.
• Pool exhaustion presents as latency, not errors: requests queue waiting for a connection, so p99 climbs while the dependency looks healthy. Instrument wait time for a connection separately from request time, or you will misdiagnose it.
• Bulkheads: separate pools per downstream dependency, so one slow service cannot consume every connection and take down endpoints that never call it.

Where does the time in p99 actually go?

The tail is almost never the same shape as the median, and knowing the mechanisms is what makes tail latency tractable:

• Cold connections. The p50 request reuses a warm connection; the p99 pays DNS + TCP + TLS. This is why "our service is fast but users say it's slow" is often a connection-reuse problem.
• TCP retransmission. A lost packet costs at minimum one RTO — often 200 ms or more — and it is invisible in application metrics. netstat -s retransmit counters and ss -ti will show it.
• Queueing. At high utilisation, queueing delay grows non-linearly — the standard result is that response time scales roughly as 1/(1-ρ), so going from 70% to 90% utilisation roughly triples queueing delay. This is why you cannot run a latency-sensitive system near capacity, and it is a very strong thing to be able to state.
• Garbage collection and JIT pauses on the server.
• Head-of-line blocking in the protocol, in a single-threaded worker, or in a message queue partition.
• Fan-out amplification. If a request calls 10 services in parallel, its latency is the maximum of 10 samples. With a 1% chance each of exceeding 1 second, the request has a ~10% chance — so p99 of your dependencies becomes p90 of your service. This is the single most important arithmetic in distributed latency, and it is the argument for hedged requests: after the p95 latency, send a duplicate request to another replica and take whichever returns first.

How should timeouts, retries and deadlines be designed?

As a budget that flows with the request, not as independent per-hop constants.

• Each layer's timeout must be shorter than its caller's remaining budget. Timeouts that grow as you go deeper are a common misconfiguration and mean the outer caller gives up while the inner work continues, wasting capacity.
• Propagate a deadline in the request (a grpc-timeout header, or your own X-Deadline), so a downstream service can refuse to start work that is already doomed. This is what context.Context does in Go and is genuinely valuable.
• Retry only idempotent operations, or non-idempotent ones protected by an idempotency key. POST is not safe to retry by default.
• Exponential backoff with full jitter, always. Without jitter, all clients retry in lockstep and the recovering service is knocked over again.
• Retry budgets — cap retries at a small percentage of total requests. Retries multiply through a call chain (3 layers × 3 attempts = up to 27× amplification) and turn a partial degradation into a full outage.
• Circuit breakers to stop calling a dependency that is failing, so you fail fast instead of burning threads and amplifying load.
• Distinguish connect timeout (short — the connection either establishes quickly or the host is unreachable) from read timeout (longer, tied to expected processing) from total deadline. A single "timeout" setting is almost always wrong.

What do proxies and load balancers change?

Every hop is a place where a header, a timeout or a buffer can change behaviour:

• L4 (TCP) balancing forwards bytes; it cannot see paths or headers, and it is fast. L7 (HTTP) terminates the connection, can route by path/header, retry, and split traffic — and it means the client's connection ends at the proxy, so keepalive and HTTP/2 must be configured on both sides. A common misconfiguration is HTTP/2 to the proxy and HTTP/1.1 with no keepalive behind it, which silently reintroduces per-request handshakes.
• Buffering. A proxy that buffers the full response breaks streaming and server-sent events (X-Accel-Buffering: no in nginx). One that buffers requests changes the timing your application sees.
• Idle timeouts at the load balancer (60 s on many cloud LBs) will kill long-lived connections and long-polling. WebSockets need explicit configuration.
• Header size limits are where a request with a large cookie or JWT starts returning 431 or 502 from the proxy, never reaching your app.
• X-Forwarded-For and Proto must be trusted only from known proxies, and your framework must be configured to know how many hops to trust — otherwise a client can spoof its IP past your rate limiter.