Project: permit domain cutover and transport recovery

Last updated: 7 Oct 20269 min read

project

Move the permit portal for district 47 from one origin pool to another. Browsers, recursive DNS resolvers, and monitoring probes disagree briefly about which address is current. A stale negative answer may hide a new verification name, and a short record TTL does not erase an older cached answer. Both origin pools must serve a valid certificate and the same session rules while traffic overlaps. The service advertises newer transport where it works, then falls back cleanly through constrained networks. Reviewers submit case mutations during the change; replayed early requests must never create a second approval. Write down the rollback decision before editing DNS.

Build contract

Inventory authoritative records, resolver observations, negative-answer lifetimes, and the old origin's latest safe retirement time. Probe both endpoints through the overlap window.
Issue and test the replacement certificate before routing users to it. Roll out HSTS scope only after every covered host serves HTTPS and recovery remains possible.
Test HTTP/2 and HTTP/3 through a normal network and a UDP-blocked network. Preserve equivalent authorization, cache rules, and error handling on fallback.
Classify early data at the gateway. Deny or retry unsafe mutations without applying side effects twice; keep an idempotency key and a server-owned operation result for approved writes.

Implementation checkpoint

function permitEarlyRequest(method, pathname) {
  const publicRead = method === "GET" && pathname === "/permit/help";
  return publicRead ? "admit-with-cache-policy" : "wait-for-handshake";
}
console.log(permitEarlyRequest("POST", "/permit/47/approve"));
// Output: wait-for-handshake

Cost and boundaries

A cutover keeps both origin pools, certificate paths, and rollback capacity alive at once. DNS response time is not the only delay: older positive answers, negative caches, connection reuse, and retrying clients all extend the overlap. The request classifier costs O(1) per admission decision for a fixed route table; the operational cost is proving that every route and downstream side effect obeys the classification. Newer transport adds a second network path to test, not a second authorization model. Track answer age, connection protocol, TLS failures, replay rejections, mutation deduplication, and origin-specific error rates without logging session secrets.

Failure drill

First, cache the old answer just before the DNS edit and query a verification name before it exists. Publish the new records, then prove the old origin still handles the stale answer and the new name stays invisible until negative caching expires. Next, remove a certificate from one origin pool in staging: the probe must fail before public routing changes. Block UDP on a client path and confirm it completes a permitted read over the fallback protocol. Finally, replay a case approval as early data and after the handshake. Only one approval may be committed; a rejected early attempt must have a clear retry path that does not assume the first request failed harmlessly.

Acceptance checks

Both origins pass certificate, host-routing, session, and private-cache checks throughout the DNS overlap.
A stale positive or negative answer changes timing but cannot strand users or hide the rollback route indefinitely.
UDP-blocked clients retain the same permitted behavior through protocol fallback.
Replayed or retried case mutations produce one committed result and a traceable response.

Common Mistakes

Removing the old origin as soon as the record is edited.
Expanding HSTS coverage before every subdomain is ready.
Using the request method alone as proof that early data is harmless.

web-tech

web-development