Project: diagnose a failing network path from client to backend

Last updated: 2 Oct 20269 min read

project

Build an isolated receipt service behind a test gateway and load balancer. Use two client networks, a Linux node where flow tracking is visible, disposable TLS names, and synthetic account data. Record the exact client address, DNS answers, negotiated protocol, TLS name and certificate, proxy peer, backend address, request size, socket age, and completed user result. Keep each fault limited to the test route; a green process probe is only one observation.

Break the route at distinct layers

Send a short read and a larger attachment through a tunnel, then lower the tunnel path MTU until only the larger operation fails. Compare retransmissions with a read-only path trace. Generate short-lived outbound flows until the test node approaches its tracking limit; preserve the counters and restore client reuse before raising capacity. Keep a client socket idle past the gateway close, then send the next read and record whether the pool retries it safely. Supply a forged forwarding address on a direct request and show that the application refuses to trust it. Repeat through the approved proxy and inspect the verified address and peer fields.

Receipt route acceptance gates
Large upload: completes across the tunnel after MTU repair
New flow: succeeds under the planned connection burst
Idle reuse: first request after a quiet period succeeds
Client address: forged direct header cannot change identity
TLS: intended name selects a valid certificate
IPv4 and IPv6: same receipt result through each family
HTTP/3: blocked UDP still leaves a working TCP request
PROXY preface: only the named load balancer can supply it

Prove identity and fallback

Use the intended TLS name and inspect the certificate before testing HTTP routing. Publish an IPv6 test address only after the Service, load balancer, firewall, and backend listener pass an end-to-end check; compare with IPv4 from a separate client network. Advertise HTTP/3 to a controlled client, record the actual negotiated protocol, then block UDP and test normal client fallback over TCP, including a client with cached alternative-service state. Enable a dedicated PROXY-aware backend port and attempt both a valid load-balancer connection and an untrusted forged preface. Recheck rate limits and audit fields on each route.

Cost and verification

Capture packet overhead after an MTU change, tracking-table memory, handshake rate after idle-timer changes, extra address-family and UDP test cost, and proxy rejection rate. The drill passes only when the user request completes and the source identity remains trustworthy under the planned fault. Record which client, protocol version, route, and failure phase produced every result; aggregated success counts hide asymmetric network failures.

Common Mistakes

Do not treat a successful small request as proof that uploads work.
Do not trust an address asserted by an arbitrary client.
Do not count protocol advertisement as a completed user request.

Connected lessons

Linux host change follow-up

nftables rollouts: apply a checked ruleset with an access recovery timer

devops

project