Networking: DNS, TLS, Routing, and Timeouts

A caller reports that a request “timed out.” That sentence describes what the caller observed, not what happened. The name might not have resolved. The server might have refused the connection, or accepted it and never answered. The certificate might have been rejected. A load balancer might have sent the request to a node that was dying. A retry might have started a second copy of work that was already running. Each of these leaves different evidence, and the only way to tell them apart is to know the path a request takes and what can fail at each step.

This article follows one HTTPS request from a client to a service and examines each step: the request path itself, DNS and certificate failures, routing and load balancing, retries and deadlines, and diagnosing at the level of bytes on the wire. The previous article, Linux Processes, Resources, and Failure Diagnosis, taught you to read how a process ended (exit status or signal), to tell CPU, memory and I/O problems apart, and to reproduce a failure under explicit limits. This article assumes that method and applies it across a network, where the same call can fail because of a machine you do not control and cannot inspect. General programming knowledge, including sockets as an abstraction, is assumed. Networking vocabulary is introduced as it is needed.

How the examples were produced matters for what they show. Everything ran locally on one macOS machine (Darwin 25.6, Python 3.12, OpenSSL 3.6) over the loopback interface, using the Python standard library and the openssl command line, with certificates created for the experiment by a throwaway test CA. Loopback has no real latency, loss or reordering, so timings show relative effects of the injected conditions and not realistic network behavior. A packet capture with tcpdump was attempted and refused for lack of permission to open a capture device, so Part 5 uses a logging relay that decodes TLS record headers from the byte stream. That is a different tool with a narrower view than a packet capture, and the article says what it cannot see. Error numbers such as errno 61 are macOS values; Linux uses different numbers for the same conditions. All code was written for this article, and all hostnames use the reserved .test domain. The listings are excerpts that explain each mechanism and are not complete programs, except the route-table listing, which is complete.

The request path: what happens between a call and a response

An HTTPS request goes through a sequence of steps, and each step has its own failure vocabulary. The following list gives the steps for a client calling https://api.svc.test/. The order is the order in which they occur for a first request on a fresh connection.

1. Name resolution. The client turns api.svc.test into one or more IP addresses by asking a DNS resolver. Possible outcomes: an address, a definitive “no such name,” a server error, or silence.
2. Routing. The client’s operating system chooses a network interface and next hop for the destination address, based on its routing table. Each router along the way repeats the decision. A missing or wrong route produces packets that never arrive, usually with no explicit error to the sender.
3. TCP connection. The client and server exchange a SYN, a SYN-ACK and an ACK (the three-way handshake). If nothing listens on the port, the destination host answers with a reset. If something drops the packet silently, the client retransmits and eventually gives up.
4. TLS handshake. The client and server negotiate parameters and the server proves its identity with a certificate. The client verifies the certificate chain, validity period and name.
5. HTTP exchange. The client sends a request and reads a response. Failures here include no response, a response cut short, and error status codes.
6. Behind the front door. The server may be a load balancer or gateway that forwards to one of several backends, each of which may call further services, repeating the same steps.

The first useful discipline is to attribute time and failure to a step. The script below times the first four steps for a request to a local TLS server, using the article’s own DNS server for resolution. In the second scenario the DNS answer is artificially delayed by 200 ms.

scenario                             dns  connect     tls  request   total   (ms)
all local, healthy                   0.2     0.1     2.3     0.3     2.9
DNS answer delayed by 200 ms       203.7     0.4     3.7     0.3   208.1

In the healthy case the whole path took under 3 ms and TLS was the largest piece, because it performs cryptographic work. In the second case DNS accounted for 203.7 of 208.1 ms, and nothing in the later steps would have shown it. A client that reports only total latency cannot distinguish these. A client library that exposes phase timings (or distributed request tracing with spans for each phase) can. The same breakdown is the right way to read a latency budget: the time allowed for a request is spent across these phases and across downstream calls, and a response latency number is only interpretable with its measurement boundary stated.

The word endpoint hides complexity here. A hostname resolves to one or more addresses, each of which may be a load balancer that represents many workers. “The endpoint is reachable” says only that some step succeeded for some path. Each of the next sections takes one step and shows how it fails.

DNS: names, caches and the failures that look like network problems

How a lookup works and what a response contains

DNS maps names to records. Each response carries a header with a response code (RCODE). RFC 1035 defines value 0 as no error, 2 as a server failure (the name server could not process the query because of a problem on its side), and 3 as a name error, meaning the queried name does not exist. A response also carries a time to live (TTL) on each record: the interval for which the record may be cached before the source should be consulted again, with zero meaning the record should not be cached. RFC 1035 also limits UDP messages to 512 bytes, with a truncation bit that tells the client to retry over TCP when a response does not fit. (Later extensions raise the UDP limit; this article uses the original rules for the simple experiment.)

Two properties follow that matter operationally. First, DNS answers are cached by intermediate resolvers and by clients, so a change takes effect only as caches expire, and the TTL is a promise about how long stale answers may be used. Second, negative answers are cached too. RFC 2308 describes caching for both “name does not exist” (NXDOMAIN) and “name exists but has no records of that type” (NODATA), with the cache lifetime taken from the minimum of the SOA record’s own TTL and its MINIMUM field. A name that was queried before it was created can therefore stay unresolvable after the record is added, until the negative cache entry expires.

To make these mechanics visible, the experiment builds a tiny DNS server and caching client that speak the real wire format (single-question A queries only). The excerpt shows how a query is encoded.

def encode_name(name):
    return b"".join(bytes([len(l)]) + l.encode() for l in name.rstrip(".").split(".")) + b"\x00"

def build_query(name, qid):
    header = struct.pack("!HHHHHH", qid, 0x0100, 1, 0, 0, 0)       # RD=1, one question
    return header + encode_name(name) + struct.pack("!HH", 1, 1)    # QTYPE=A, QCLASS=IN

The client below was run against a server with four kinds of names: a normal record with TTL 30, a name that does not exist, a name that returns a server failure, and a name whose queries are silently dropped. Times for cache behavior use an injected clock so that TTL expiry is deterministic; the timing column is real. Its output:

api.svc.test       -> 10.0.0.7   server (attempt 1, ttl 30)                   server_queries+=1 elapsed=   0.4 ms
api.svc.test       -> 10.0.0.7   cache                                        server_queries+=0 elapsed=   0.0 ms
api.svc.test       -> 10.0.0.7   server (attempt 1, ttl 30)                   server_queries+=1 elapsed=   0.2 ms
apii.svc.test      -> NXDOMAIN   server (attempt 1)                           server_queries+=1 elapsed=   0.2 ms
apii.svc.test      -> NXDOMAIN   cache                                        server_queries+=0 elapsed=   0.0 ms
apii.svc.test      -> NXDOMAIN   server (attempt 1)                           server_queries+=1 elapsed=   0.2 ms
broken.svc.test    -> RCODE 2    server (attempt 1)                           server_queries+=1 elapsed=   0.3 ms
silent.svc.test    -> TIMEOUT    no answer after 3 attempts of 0.3s           server_queries+=3 elapsed= 901.5 ms
wire bytes of the first query the server saw: 00 01 01 00 00 01 00 00 00 00 00 00 03 61 70 69 03 73 76 63 04 74 65 73 74 00 00 01 00 01

Reading it row by row. The first lookup went to the server (one query). The second, 10 seconds later, was served from cache and no query was sent. The third, 31 seconds after the first, found the 30-second TTL expired and asked again. The misspelled name apii.svc.test returned NXDOMAIN, and the repeat two seconds later was answered from the negative cache (the client uses a 5-second negative TTL here, a value chosen for the experiment). After the negative entry expired, the client asked again. The SERVFAIL row shows RCODE 2 as a response that is neither an answer nor a statement that the name does not exist. The last row is the one that costs time: the server never answered, the client waited 0.3 seconds per attempt for three attempts, and the call took 901.5 ms, 3 times the per-attempt timeout. Both numbers are the client’s own configuration, which is the point: the duration of a DNS outage as seen by the caller is attempts × timeout of whatever resolver settings it uses.

The final line prints the first query’s bytes as the server received them. Decoded: the first two bytes 00 01 are the query ID, 01 00 is the flags field with only the recursion-desired bit set, and 00 01 00 00 00 00 00 00 gives one question and no other records. Then comes the name as length-prefixed labels (03 61 70 69 is the 3-byte label api, 03 73 76 63 is svc, 04 74 65 73 74 is test) ending in 00, followed by type 1 (A) and class 1 (IN). Being able to read this is what makes packet-level diagnosis possible later.

DNS failures and how they present

What happened	What the caller sees	How to tell
Name does not exist (RCODE 3)	Immediate resolution error, no connection attempted	Query the name directly and read the response code; check for typos and for negative caching of a name created recently
Server failure (RCODE 2)	Immediate error that is not “not found”	The resolver or authoritative server has a problem; try the authoritative server and a second resolver
No answer	Delay of attempts times timeout, then an error	Compare resolver timeout settings with the observed delay; a delay that equals a round multiple of a configured timeout is a signature
Stale record (TTL not yet expired)	Connections go to the old address	Compare the answer from the resolver with the authoritative answer and read the remaining TTL
Different answers from different resolvers	Intermittent failures depending on which resolver the client used	Query each resolver separately; the cache state and the record sets can differ

The common lesson is that a DNS problem can masquerade as a connection problem. A service whose address changed during a migration appears to fail for exactly the clients whose caches have not expired. The fix cannot be derived from the service’s logs, because the requests never arrive.

Certificates: why a reachable server can still be rejected

TLS has two jobs: encrypting the connection and authenticating the server, and the second is where certificate failures come from. In the TLS 1.3 handshake described in RFC 8446, the client sends a ClientHello with the versions, cipher suites and key shares it supports. The server answers with a ServerHello, after which all further handshake messages are encrypted. The server then sends its parameters, its certificate, a proof that it holds the certificate’s private key, and a Finished message. The client’s job after receiving the certificate is to decide whether to trust it, and the decision has several independent parts.

• Chain. The certificate must be signed by an issuer, and that issuer’s certificate by another, until the chain reaches a trust anchor the client already holds. RFC 5280 describes path validation as verifying each certificate’s signature using the issuer’s public key, confirming each certificate has not expired, and checking revocation status, ending at a trust anchor. If the client does not hold the issuing CA, validation fails whatever else is true.
• Validity period. A certificate carries notBefore and notAfter dates, and RFC 5280 defines the validity period as inclusive of both. A certificate outside the period is rejected, which makes expiry a calendar-driven outage that happens to every certificate that nobody renews.
• Name. The certificate must be valid for the name the client intended to reach. RFC 9525 describes the client building a reference identifier from the name it was asked to connect to and comparing it against identifiers in the certificate. It states that the Common Name field must not be used to identify a service, and that a wildcard may appear only as the entire leftmost label, with at most one wildcard character. The intended name is the one the user or configuration supplied, not the address that DNS returned.

Revocation checking, the third item in the chain step, depends on infrastructure outside the handshake and varies by client; the experiment below does not exercise it and this article makes no claim about it.

To see the failures, the experiment creates a private test CA with the openssl command line and issues certificates for api.svc.test: a good one, one valid only for other.svc.test, one with a validity period ending in the past, and one issued by a second CA the client does not trust. Every certificate deliberately has a Common Name that matches nothing, so the successful case also shows that the name check does not rely on it. The client verifies against the test CA and requests the name api.svc.test while connecting to the loopback address, which is what a real client does after DNS resolves. The core of the client is below.

def try_connect(label, certname, host, cafile="pki/ca.pem"):
    ready = []; serve(certname, ready); port = ready[0]
    ctx = ssl.create_default_context(cafile=cafile) if cafile else ssl.create_default_context()
    try:
        with socket.create_connection(("127.0.0.1", port), timeout=2) as s:
            with ctx.wrap_socket(s, server_hostname=host) as t:
                t.sendall(b"GET / HTTP/1.1\r\nHost: x\r\n\r\n"); body = t.recv(1024).split(b"\r\n\r\n")[1]
                print(f"{label:46s} OK  protocol={t.version()} body={body!r}")
    except ssl.SSLCertVerificationError as e:
        print(f"{label:46s} FAIL SSLCertVerificationError code={e.verify_code}: {e.verify_message}")
    except Exception as e:
        print(f"{label:46s} FAIL {type(e).__name__}: {e}")

Output of the six cases:

valid cert, connect as api.svc.test            OK  protocol=TLSv1.3 body=b'ok'
cert for other.svc.test, connect as api.svc.test FAIL SSLCertVerificationError code=62: Hostname mismatch, certificate is not valid for 'api.svc.test'.
expired cert (notAfter 2026-02-01)             FAIL SSLCertVerificationError code=10: certificate has expired
cert issued by a CA the client does not trust  FAIL SSLCertVerificationError code=20: unable to get local issuer certificate
valid cert, but verifying against 127.0.0.1    FAIL SSLCertVerificationError code=64: IP address mismatch, certificate is not valid for '127.0.0.1'.
valid cert, client has no copy of our test CA  FAIL SSLCertVerificationError code=20: unable to get local issuer certificate

Each failure has a distinct message, and each corresponds to a distinct fix. A hostname mismatch means the certificate was issued for the wrong name or the client used the wrong name. Expiry means renewal failed or was not scheduled, and the fix is renewal plus a process that alerts well before notAfter. An unknown issuer means the client’s trust store lacks the CA, which happens when a private CA is used and the trust store was not updated, or when a server does not send the intermediate certificate. The fourth failure is worth reading closely: verifying against 127.0.0.1 fails because the certificate lists only a DNS name, and the client was asked to verify an IP address, which RFC 9525 treats as a separate kind of identifier. A client that connects by IP address to a service whose certificate names only DNS identifiers will fail even though the server is correct. The last row shows that the same valid certificate fails for a client that lacks the test CA.

Two properties of certificate failures make them hard to notice in advance. First, they are binary and sudden: a certificate is valid until the moment it is not, and every client fails at once. Second, they can be partial: a certificate rotated on only some servers behind a load balancer produces failures for a fraction of requests that matches the fraction of servers with the old certificate, which looks like random flakiness. An inventory of certificates with owners and expiry dates, monitored from the client’s side (a probe that performs a full verification against the real name), catches the failure modes above. A check that only confirms the port is open does not.

Routing and load balancing: how a request reaches a particular machine

Routing decides the next hop

A routing table maps destination prefixes to next hops. When several entries match a destination, the most specific one wins. RFC 4632 states that forwarding on the Internet is done on a longest-match basis, and describes 0.0.0.0/0 as the default route covering the whole IPv4 address space. The following program applies that rule to an invented route table.

import ipaddress

# An illustrative route table (made-up addresses from the documentation/private ranges).
ROUTES = [
    ("0.0.0.0/0",     "internet-gateway"),
    ("10.0.0.0/16",   "local"),
    ("10.0.8.0/24",   "firewall-appliance"),
    ("10.1.0.0/16",   "peering-link"),
]
TABLE = [(ipaddress.ip_network(p), t) for p, t in ROUTES]

def lookup(dst):
    ip = ipaddress.ip_address(dst)
    matches = [(n, t) for n, t in TABLE if ip in n]
    best = max(matches, key=lambda m: m[0].prefixlen)       # longest prefix wins
    return best, [str(n) for n, _ in matches]

for dst in ["10.0.3.9", "10.0.8.40", "10.1.4.4", "10.2.0.1", "198.51.100.7"]:
    (net, target), matched = lookup(dst)
    print(f"{dst:15s} matches {matched} -> uses {net} via {target}")

10.0.3.9        matches ['0.0.0.0/0', '10.0.0.0/16'] -> uses 10.0.0.0/16 via local
10.0.8.40       matches ['0.0.0.0/0', '10.0.0.0/16', '10.0.8.0/24'] -> uses 10.0.8.0/24 via firewall-appliance
10.1.4.4        matches ['0.0.0.0/0', '10.1.0.0/16'] -> uses 10.1.0.0/16 via peering-link
10.2.0.1        matches ['0.0.0.0/0'] -> uses 0.0.0.0/0 via internet-gateway
198.51.100.7    matches ['0.0.0.0/0'] -> uses 0.0.0.0/0 via internet-gateway

Address 10.0.8.40 matches three entries, including the default route and the 10.0.0.0/16 local range, but the /24 entry is the most specific, so traffic goes to the firewall appliance. This is how a small, specific route can silently divert traffic that a broader route would have kept local, and why “the route exists” is not enough to explain a path. In a cloud environment, virtual networks express the same ideas as subnets, route tables and security rules, and the glossary entry separates reachability (does routing make a path possible) from permission (does a security rule allow traffic along it). When a connection hangs with no error, both are candidates, and they are diagnosed differently.

Load balancing chooses among equivalent backends

A load balancer sits in front of several backends and chooses one per request or per connection. The policy matters because backends are rarely equal in practice. Three policies are common. Round robin cycles through backends in order. Random picks one uniformly. Least-outstanding (also called least request) picks the backend with the fewest requests currently in flight. Envoy’s documentation describes its least-request policy, when weights are equal, as choosing two hosts at random and taking the one with fewer active requests (the power of two choices), which keeps the cost per decision constant and resists many clients choosing the same node at once. The same documentation says random can perform better than round robin when no health checking is configured, because it avoids a bias when hosts fail. Consistent hashing policies exist for cases where the same key should reach the same backend.

The experiment starts three real HTTP backends on loopback: two that take 10 ms per request and one with an injected 150 ms delay (a stand-in for a node that is degraded). Eight concurrent client threads send 160 requests through a small client-side balancer. The core of the balancer is the choice function.

class Balancer:
    def __init__(self, ports, policy, eject_after=0, retry=False):
        self.ports, self.policy, self.eject_after, self.retry = ports, policy, eject_after, retry
        self.rr = itertools.cycle(range(len(ports))); self.inflight = [0] * len(ports); self.fails = [0] * len(ports)
        self.ejected = set(); self.lock = threading.Lock(); self.rng = random.Random(1); self.hits = [0] * len(ports)
    def pick(self, exclude=()):
        with self.lock:
            cand = [i for i in range(len(self.ports)) if i not in self.ejected and i not in exclude] or [i for i in range(len(self.ports)) if i not in exclude]
            if self.policy == "round_robin":
                while True:
                    i = next(self.rr)
                    if i in cand: break
            elif self.policy == "least_outstanding": i = min(cand, key=lambda j: self.inflight[j])
            else: i = self.rng.choice(cand)
            self.inflight[i] += 1; self.hits[i] += 1; return i

backends: two at 10 ms, one at 150 ms (injected delay); 160 requests, 8 concurrent clients
round robin                              total= 1.29s p50=  14.4ms p95= 157.6ms errors=  0 per-backend requests=[54, 53, 53]
random                                   total= 1.42s p50=  14.1ms p95= 155.4ms errors=  0 per-backend requests=[44, 54, 62]
least outstanding requests               total= 0.48s p50=  14.0ms p95=  19.2ms errors=  0 per-backend requests=[78, 76, 6]

Round robin and random gave the slow node about a third of the requests, so about a third of requests took around 150 ms: p95 was 157.6 ms and 155.4 ms, and the run took 1.29 and 1.42 seconds. Least-outstanding gave the slow node only 6 of 160 requests (it was usually busy), and p95 fell to 19.2 ms with the run finishing in 0.48 seconds. The mechanism is that a slow node accumulates in-flight requests, which the policy observes. The limits of the demonstration need stating. The backends here differ in a fixed way that is visible through queue length; real degradation can be intermittent; the client counts only its own in-flight requests, which a fleet of balancers cannot see globally; and a single short run on one machine is not a benchmark. The experiment supports the claim that a policy which observes load can avoid a slow node that a blind policy keeps feeding, and nothing stronger.

Health tracking removes failed backends, and retries cover the gap

Selection policy cannot help if a backend is simply gone. Health tracking comes in two forms. Active health checking sends probes to each backend on an interval and marks hosts unhealthy or healthy after configurable counts of failures or successes. Passive health checking (Envoy calls it outlier detection) watches real traffic and ejects a host after consecutive errors, for a time that grows with repeated ejection and is bounded by a maximum percentage of the cluster. Envoy’s documentation notes that the two are commonly combined, with a long active interval to limit probe traffic while passive detection reacts quickly. The second half of the experiment replaces the slow node with a port where nothing listens.

backends: two healthy, one with nothing listening (connection refused)
round robin, no health tracking, no retry total= 0.20s p50=  12.4ms p95=  17.6ms errors= 53 per-backend requests=[54, 53, 53]
round robin, retry once on another       total= 0.34s p50=  15.4ms p95=  27.6ms errors=  0 per-backend requests=[80, 80, 65]
round robin, eject after 2 failures      total= 0.29s p50=  13.8ms p95=  16.6ms errors=  3 per-backend requests=[79, 78, 3]

With no health tracking and no retry, every third request hit the dead port: 53 of 160 failed (the dead port received 53). Retrying once on a different backend made all 160 succeed, at the cost of extra work: the per-backend counts sum to 225 attempts, because 65 attempts hit the dead port and were repeated elsewhere. Ejecting the dead backend after two consecutive failures cut errors to 3, which are the failures that occurred before ejection took effect with eight requests in flight, and attempts to the dead port to 3. The comparison shows that retry hides the failure from callers while continuing to waste attempts on the broken node, and ejection stops the waste but lets a few requests fail first. In practice the two are combined. Both rely on the failure being retried safely: a refused connection is safe because nothing was processed, which is not true of every failure, as the next section explains.

A gateway in front of services adds another layer to this path, handling routing, credential checks, rate limits and telemetry, and each of those is a place where a request can be refused with a status code that has nothing to do with the service itself. When diagnosing, identify which layer produced a response: RFC 9110 distinguishes 502 (a gateway received an invalid response from upstream), 503 (the server is temporarily unable to handle the request) and 504 (a gateway did not get a timely response from upstream), and these three tell you the failure is upstream of the component that answered.

Timeouts, retries and deadlines

A timeout is a symptom with several different causes

The word timeout covers several situations that should be separated. The following program creates four conditions on loopback and reports how a client with a 300 ms timeout experiences each.

nothing listening on the port      after    2.2 ms -> ConnectionRefusedError: [Errno 61] Connection refused
accepts, never answers             after  300.5 ms -> TimeoutError: timed out
accepts, then closes with RST      after    0.8 ms -> ConnectionResetError: [Errno 54] Connection reset by peer
answers, body shorter than declared after    0.7 ms -> IncompleteRead: IncompleteRead(5 bytes read, 95 more expected)

A port with nothing listening is refused immediately (here in 2.2 ms), because the destination host answered with a reset; RFC 9293 requires a host with no matching connection to send a reset in response to an incoming segment. That immediate failure says the host is up and the service is not listening. A server that accepts the connection and never answers produces a timeout exactly at the client’s limit, and from the client’s side it is indistinguishable from a network that dropped the packets, except that the connection was established. A reset sent after the request arrives is a connection torn down by the peer. A response shorter than its declared length shows up as an incomplete read: the server promised 100 bytes and sent 5. Each is a different diagnosis, and a client that logs only “timeout” or “request failed” has lost the distinction. The log line should record the phase (connect, TLS, waiting for the first byte, reading the body), the error class, and the elapsed time.

TCP also retransmits unacknowledged data using a retransmission timeout that RFC 9293 requires to be computed by the algorithm in RFC 6298. The consequence is that a packet loss costs at least a retransmission timeout of delay, which is typically far longer than a normal round trip, so a lossy path appears as latency spikes without errors. That behavior could not be demonstrated on loopback.

Retries: when they help and how they hurt

A retry is a bet that a failure was transient. It wins when the first failure was caused by something that has since changed, such as a connection to a node that was removed, and loses when the failure was caused by load, in which case the retry adds load. Three conditions govern whether a retry is safe and useful.

• The operation is safe to repeat. RFC 9110 defines an idempotent method as one where multiple identical requests have the same effect as a single request and lists GET, HEAD, PUT, DELETE, OPTIONS and TRACE as idempotent. It also notes that some requests can be automatically retried after a connection failure. A POST that creates an order is not idempotent by default; repeating it can create two orders. Making it safe takes an idempotency mechanism such as a client-supplied key that the server uses to recognize a repeat. When the caller did not see the response, it cannot know whether the work happened. This ambiguity is the core of a partial failure.
• The retries are spaced and bounded. Retrying immediately turns a brief overload into sustained overload. Exponential backoff lengthens the wait after each failure, and jitter randomizes the wait so that many clients do not retry in lockstep. If the server sends a Retry-After value, the retry-after and backoff entry describes treating it as a floor for the wait. An attempt count, an overall deadline and a retry budget bound the total work.
• Retries do not multiply across layers. The Amazon Builders’ Library describes the effect: when each of several stacked layers retries, load on the lowest layer multiplies, and it recommends retrying at a single point in the stack, together with token-bucket retry budgets.

The multiplication is easy to underestimate, so the experiment measures it. Three layers (a client, service A and service B) call a dependency C that always returns 503. Each layer treats a 5xx as retryable and makes up to a configured number of attempts, with no backoff so that the run is quick. The script counts calls that reach C for one user request. The retry loop at each layer is below.

def call(port, attempts, timeout=2.0, headers=None):
    """Return the status of the last attempt; retry on 5xx or connection error up to `attempts` times, no backoff (kept short)."""
    status = None
    for _ in range(attempts):
        try:
            c = http.client.HTTPConnection("127.0.0.1", port, timeout=timeout); c.request("GET", "/", headers=headers or {})
            status = c.getresponse().status; c.close()
            if status < 500: return status
        except OSError: status = 599
    return status

Experiment 1: one user request, dependency C always returns 503; count calls that reach C
  no retries anywhere                    attempts(client,A,B)=(1, 1, 1)  calls reaching C = 1
  retry only at the edge (3 attempts)    attempts(client,A,B)=(3, 1, 1)  calls reaching C = 3
  every layer makes 3 attempts           attempts(client,A,B)=(3, 3, 3)  calls reaching C = 27

With no retries, one user request produced one call to C. With 3 attempts at the edge only, it produced 3. With 3 attempts at every layer it produced \( 3 \times 3 \times 3 = 27 \) calls. The general rule for these assumptions (every layer makes \( a \) attempts in total and the failure persists) is that the bottom layer receives \( a^n \) calls from \( n \) layers for each user request. The assumption of a permanent failure is the worst case. For a failure that clears after a short time, most requests succeed on a retry and the multiplication applies only to the requests that keep failing. But the dependency that fails because it is overloaded is the case where a persistent failure is likely, and the multiplied load is what keeps it down. This is the mechanism behind a cascading failure, in which the system’s own reaction to a failure increases the load that caused it. Load shedding at the overloaded layer is the complementary protection, because it rejects work cheaply instead of letting retries pile up.

Deadlines propagate; timeouts alone do not

A per-call timeout bounds one attempt. A deadline bounds the whole operation, and it is a point in time after which the original caller is no longer waiting. The gRPC documentation distinguishes the two: a timeout is a duration, a deadline is the time at which the caller stops waiting, and a server that makes downstream calls should honor the original deadline so that work is not done for a caller who has gone. It also describes servers canceling a call once the client’s deadline has passed, and notes that implementations convert deadlines to remaining timeouts when passing them on, which avoids depending on synchronized clocks between machines.

The second experiment shows the effect. A user gives up after 500 ms. Service A calls C, which takes 350 ms and is never canceled. A allows 300 ms per attempt and up to 5 attempts, so each attempt times out and is retried. In one run A ignores the user’s deadline; in the other, the client passes its remaining budget in a header and A shortens each attempt to the time left and does not start an attempt when under 50 ms remain. The deadline-aware loop is below.

def deadline_run(propagate):
    started = {"C": 0}; lock = threading.Lock()
    def c(h):
        with lock: started["C"] += 1
        time.sleep(0.35); return 200                           # C needs 350 ms; it is never cancelled
    pc = serve(c)
    finished_at = {}
    def a(h):
        t_start = time.perf_counter()
        budget = float(h.headers.get("X-Deadline-Ms", "1e9")) / 1000 if propagate else 1e9
        status = 504
        for _ in range(5):                                      # up to 5 attempts, 300 ms per attempt
            remaining = budget - (time.perf_counter() - t_start)
            if remaining < 0.05: break                          # no time left for a useful attempt
            try:
                cn = http.client.HTTPConnection("127.0.0.1", pc, timeout=min(0.3, remaining)); cn.request("GET", "/"); status = cn.getresponse().status; cn.close(); break
            except OSError: status = 504
        finished_at["A"] = time.perf_counter() - t_start
        return status

Experiment 2: user deadline 500 ms; C takes 350 ms; A allows 300 ms per attempt, up to 5 attempts
  deadline propagated=False -> client gave up at 0.5 s; C executions started=5; A kept working for 1.51 s
  deadline propagated=True  -> client gave up at 0.5 s; C executions started=2; A kept working for 0.50 s

Without propagation, A kept working for 1.51 seconds after the user left, starting 5 executions at C, all of them wasted because no one was waiting for the results. With propagation, A stopped at 0.50 seconds and C started 2 executions. The experiment makes two further points. Each of C’s executions continued after A’s client side timed out, because nothing told C to stop; a timeout abandons waiting and does not cancel work, and that is why gRPC’s server-side cancellation matters. And the deadline-aware version still started two executions, since each one needed more time than the budget allowed. A deadline reduces waste, and cannot make a slow dependency fast. A stack with per-layer timeouts that sum to more than the user’s patience is a design error visible in this data: A’s 5 attempts at 300 ms is 1.5 seconds of budget against 0.5 seconds of user patience. Timeouts should be chosen from the downstream latency distribution (the Builders’ Library suggests selecting from a high percentile of that distribution) and checked against the caller’s deadline.

Packet-level diagnosis: reading what actually crossed the wire

When logs, metrics and status codes disagree or are missing, the bytes are the ground truth. A packet capture records packets as seen at a point on the path and lets you see the SYN that was never answered, the retransmissions, the reset, the order of messages and the gaps between them. The standard tool is tcpdump or Wireshark, with a command such as tcpdump -i <interface> host <address> and port 443 -w capture.pcap. That command was not run for this article: opening a capture device requires privileges that were not available, and the attempt returned a permission error. Instead, the experiment places a logging relay between a client and the TLS server. It forwards bytes unchanged and decodes TLS record headers (content type, version, length) and, where the content is not encrypted, the handshake message type. This is weaker than a capture: it sees a reassembled byte stream, not individual TCP segments, acknowledgements, retransmissions or timing at packet granularity. The parsing core is below.

def sni_from_client_hello(body):
    try:
        p = 4 + 2 + 32; p += 1 + body[p]                        # skip handshake header, version, random, session id
        p += 2 + struct.unpack("!H", body[p:p+2])[0]            # cipher suites
        p += 1 + body[p]                                        # compression methods
        end = p + 2 + struct.unpack("!H", body[p:p+2])[0]; p += 2
        while p < end:
            et, el = struct.unpack("!HH", body[p:p+4])
            if et == 0: return body[p+9:p+4+el].decode()       # server_name extension
            p += 4 + el
    except Exception: pass
    return None

def pump(src, dst, label, t0, log):
    buf = b""
    try:
        while True:
            data = src.recv(65536)
            if not data:
                log.append(f"{1000*(time.perf_counter()-t0):7.1f} ms  {label}: connection closed by sender (FIN)")
                try: dst.shutdown(socket.SHUT_WR)
                except OSError: pass
                return
            dst.sendall(data); buf += data
            while len(buf) >= 5:
                ct, ver, ln = struct.unpack("!BHH", buf[:5])
                if len(buf) < 5 + ln: break
                rec, buf = buf[5:5+ln], buf[5+ln:]
                note = ""
                if ct == 22 and rec and rec[0] in HT:
                    note = f" -> {HT[rec[0]]}"
                    if rec[0] == 1: note += f" (SNI={sni_from_client_hello(rec)})"
                if ct == 21 and ln == 2: note = f" -> alert level={rec[0]} description={rec[1]}"
                log.append(f"{1000*(time.perf_counter()-t0):7.1f} ms  {label}: TLS record type={ct}({CT.get(ct,'?')}) version=0x{ver:04x} length={ln}{note}")
    except OSError as e:
        log.append(f"{1000*(time.perf_counter()-t0):7.1f} ms  {label}: {type(e).__name__}")

The log of one successful connection and one that fails because the certificate is for the wrong name follows. Times are milliseconds since the relay accepted the connection.

=== valid certificate: OK
      0.4 ms  client->server: TLS record type=22(handshake) version=0x0301 length=1512 -> ClientHello (SNI=api.svc.test)
      2.0 ms  server->client: TLS record type=22(handshake) version=0x0303 length=1210 -> ServerHello
      2.0 ms  server->client: TLS record type=20(change_cipher_spec) version=0x0303 length=1
      2.0 ms  server->client: TLS record type=23(application_data) version=0x0303 length=23
      2.0 ms  server->client: TLS record type=23(application_data) version=0x0303 length=834
      2.0 ms  server->client: TLS record type=23(application_data) version=0x0303 length=281
      2.0 ms  server->client: TLS record type=23(application_data) version=0x0303 length=69
      2.6 ms  client->server: TLS record type=20(change_cipher_spec) version=0x0303 length=1
      2.6 ms  client->server: TLS record type=23(application_data) version=0x0303 length=69
      2.7 ms  client->server: TLS record type=23(application_data) version=0x0303 length=44
      2.7 ms  server->client: TLS record type=23(application_data) version=0x0303 length=250
      2.8 ms  server->client: TLS record type=23(application_data) version=0x0303 length=250
      2.9 ms  server->client: TLS record type=23(application_data) version=0x0303 length=76
      2.9 ms  server->client: connection closed by sender (FIN)
      3.0 ms  client->server: connection closed by sender (FIN)
=== certificate for the wrong name: client raised: Hostname mismatch, certificate is not valid for 'api.svc.test'.
      0.4 ms  client->server: TLS record type=22(handshake) version=0x0301 length=1512 -> ClientHello (SNI=api.svc.test)
      2.0 ms  server->client: TLS record type=22(handshake) version=0x0303 length=1210 -> ServerHello
      2.0 ms  server->client: TLS record type=20(change_cipher_spec) version=0x0303 length=1
      2.0 ms  server->client: TLS record type=23(application_data) version=0x0303 length=23
      2.0 ms  server->client: TLS record type=23(application_data) version=0x0303 length=836
      2.0 ms  server->client: TLS record type=23(application_data) version=0x0303 length=281
      2.0 ms  server->client: TLS record type=23(application_data) version=0x0303 length=69
      2.5 ms  client->server: TLS record type=21(alert) version=0x0303 length=2 -> alert level=2 description=42
      2.6 ms  client->server: ConnectionResetError
      2.6 ms  server->client: connection closed by sender (FIN)

Reading the successful connection. The client’s first record is a handshake record containing a ClientHello, and the relay decoded the server name indication (SNI) extension: api.svc.test. The SNI tells the server which name the client intends to reach, which lets one address serve many certificates. The server’s reply is a ServerHello, and every record after that appears as application_data (type 23) with no decodable content, consistent with RFC 8446’s statement that handshake messages after ServerHello are encrypted. A change_cipher_spec record (type 20) appears for compatibility and carries no meaning in TLS 1.3. The relay therefore cannot show the certificate, and a capture cannot either without the session keys. Then application data flows both ways, and both sides close (FIN).

Reading the failing connection. The server’s side is identical, since it has no way to know yet that the client will reject its certificate. Then the client sends a record of type 21 (alert) with length 2, and the relay decoded its two bytes as level 2 and description 42. RFC 8446 defines level 2 as fatal, requiring immediate termination of the connection. The IANA TLS alerts registry assigns 42 to bad_certificate. The client then closes and the relay logs a ConnectionResetError on that leg. The order is the evidence: the handshake proceeded normally up to the point where the client examined the certificate, and the client was the one that ended it. That is the signature of a client-side certificate rejection, and it tells you where to look (the certificate and the client’s trust and name settings) without access to the client’s logs. One observation specific to this run: the alert was visible as a plain record, which the experiment recorded without a claim that this holds for other clients and versions. The alert description shows only the category, and the client’s error message gave the specific reason, so diagnosing the cause still needs the client’s own error text.

A method for reading a capture

1. Establish the vantage point. A capture on the client, the server or a middlebox shows different things. A packet visible on the server and absent on the client’s side tells you where it was lost or rewritten.
2. Find the connection and classify its start. Look for the SYN. A SYN with no reply and repeated SYNs at growing intervals points to dropping by something on the path or a full accept queue. A reset in reply to a SYN points to a host with nothing listening, or a filter that rejects.
3. Check the handshake order. For TLS, a ClientHello with no ServerHello means the server did not respond or the traffic was blocked. A fatal alert immediately after the server’s certificate points to the client rejecting it.
4. Look at gaps and retransmissions. A long gap between a request and the first response byte is server time; retransmitted segments with growing spacing are network loss or an unresponsive peer.
5. Identify who closed and how. A graceful close (FIN) and an abrupt one (RST) mean different things. A reset after the request is sent points to the peer or a middlebox discarding the connection.
6. Correlate with logs by time and connection. A capture is a timeline. The client’s log of the same request should explain each gap or the capture shows where the logs are wrong.

Two cautions apply. A capture of encrypted traffic shows timing, sizes and record types, but not content, so TLS troubleshooting depends on the client’s own error output and on the handshake-level view above. And captures can contain sensitive data, so handle them like any other record of user traffic and do not share them without checking what they hold.

Putting the path back together

The symptom “timeout” now decomposes into a sequence of questions, and each has a cheap first check.

Step	Question	First evidence	Typical fix direction
DNS	Did the name resolve to the right address, quickly?	Direct query and response code; resolver timeout settings; remaining TTL	Correct the record, wait out or shorten TTLs, fix the resolver
Route	Is there a path, and is it the intended one?	Route lookup for the destination; most specific matching entry	Fix the route or the more specific entry that diverts traffic
TCP	Refused, silent or reset?	Elapsed time and error class of the connect	Start the service, open the filter, or fix the accept queue
TLS	Was the certificate accepted?	Client verification error text; alert description	Renew, reissue for the right name, distribute the CA, send the intermediate
Balancer	Which backend served it, and was it healthy?	Per-backend counts and error rates; health state	Change the policy, add health tracking, remove the bad node
Retries	How many attempts reached the dependency?	Attempts per user request at each layer	Retry at one layer, add budgets, backoff and jitter, ensure idempotency
Deadlines	Is anyone still waiting for this work?	Remaining budget at each hop; work continuing after the client left	Propagate deadlines and cancel downstream work

The limits of this article are those of its tools. It ran on one machine over loopback, so it showed mechanisms and not the performance of any real network, and the figures are from single runs. Loss, reordering, middleboxes, IPv6 behavior, revocation checking, HTTP/2 and HTTP/3, and cloud provider load balancers were not exercised. Packet captures were described and not performed. The experiments support narrow claims: that each failure has a distinguishable signature in the evidence, that a load-aware policy can avoid a degraded node a blind one feeds, that retries multiply across layers, and that deadlines reduce wasted work. Whether those effects matter in a particular system is a measurement to make there.

The next article, Cloud Accounts, Identity, and Landing Zones, moves from how requests flow to who is allowed to do what and where: the account boundaries, identity and network baselines within which these services run.

References

• RFC 1035, Domain Names – Implementation and Specification
• RFC 2308, Negative Caching of DNS Queries (DNS NCACHE)
• RFC 4632, Classless Inter-domain Routing (CIDR)
• RFC 5280, Internet X.509 Public Key Infrastructure Certificate and CRL Profile
• RFC 8446, The Transport Layer Security (TLS) Protocol Version 1.3
• RFC 9110, HTTP Semantics
• RFC 9293, Transmission Control Protocol (TCP)
• RFC 9525, Service Identity in TLS
• IANA, Transport Layer Security (TLS) Parameters, TLS Alerts registry
• Envoy documentation, Load Balancers
• Envoy documentation, Outlier detection
• Envoy documentation, Health checking
• gRPC documentation, Deadlines
• Amazon Builders’ Library, Timeouts, retries, and backoff with jitter