DNS controls future address resolution, not every connection already held by a client pool. Long-lived HTTP/2 channels may continue sending streams to an old endpoint until the channel drains, fails, or is deliberately rotated. A proxy can also retain upstream connections while service discovery changes. A rollout can therefore serve mixed backend versions even after DNS and target lists look correct.
Make this comfortable
Client connection pools: retire old endpoints after a DNS or rollout change
Operational decision
A claims API moves from one internal gateway to another. Before the cutover, record the client resolver result, active connections by peer address, connection age, request version header, and in-flight work. Change the destination in an isolated environment and watch both address resolution and existing sockets. Ask the old endpoint to drain; new streams should use the replacement while accepted work finishes. If a client keeps a channel indefinitely, apply a bounded connection lifetime or an explicit pool rotation supported by its library, then test it under load. Do not restart every client blindly: a simultaneous reconnect storm can overwhelm DNS, TLS termination, and the new gateway. Track retries by operation ID so a request that was accepted before disconnect is reconciled before repetition. Confirm that the last old-version response disappears within the cutover contract, not merely that the latest lookup returns the new address.
Claims endpoint cutover evidence
Resolver answers: old and new addresses over time
Live sockets: peer address, age, in-flight count
Old endpoint: drain signalled and last stream completed
New endpoint: handshake and request success under load
Client pool: rotation rule and reconnect rate bounded
Acceptance: no unexplained old-version responses or duplicate effectsCost and verification
Short connection lifetimes speed convergence but increase handshakes, CPU, and latency. Long lifetimes conserve work but lengthen a mixed-version window. A synchronized restart can make the new endpoint fail despite enough steady-state capacity. Measure churn, TLS handshakes, DNS queries, active streams, and business retries while shifting traffic. Tune the client library and proxy separately because each may keep its own pool.
Common Mistakes
- Do not assume a short DNS TTL closes an established socket.
- Do not repeat an ambiguous write solely because the old connection reset.
- Do not rotate every client pool at once without a reconnect budget.
Connected lessons
- DevOps: delivery, infrastructure, and reliable operations
- DNS cutovers: budget for resolver caches and mixed destinations
- Connection draining: let in-flight work finish while new traffic moves
- NAT port pressure: find the shared outbound ceiling
- Idempotency keys: reconcile an accepted write before repeating it
Practice and check
Advanced follow-up
devops
operations