Project: rotate service trust and contain an exposed TLS key

Last updated: 5 Oct 202610 min read

project

Build a disposable internal order API with a private CA, two client implementations, a gateway, and a second listener representing the recovery region. Keep its keys and test certificates out of version control. The exercise passes only when both client classes validate the served chain and name, a new root becomes trusted before new leaves appear, every listener serves the new identity, an unrelated mTLS caller is denied, and a compromised test key ceases to be accepted on the paths under test.

Map names, chains, and clients

Record the intended external hostname, internal upstream name, certificate SANs, presented intermediates, and client trust anchors. Open fresh handshakes from a clean container and a second client stack against both listeners. In a disposable test, omit an intermediate, send wrong SNI, and replace the upstream certificate with a same-CA leaf for another service. Confirm each failure appears at the correct hop. Do not count a cached laptop handshake as coverage for the container or recovery listener.

Overlap and rotate trust

Create an incoming CA and distribute a versioned bundle containing current and incoming roots to the clients. Verify the process actually loaded the bundle before issuing a new server leaf. Rotate the leaf and private key, then compare the Secret fingerprint with fresh handshakes to every Pod; test whether a long-lived connection survives or must drain. Remove the old root only after clients have migrated, and prove an old-only chain is rejected. Keep client trust data separate from the Secret containing the server private key.

Trust rotation acceptance
Clean clients build the intended chain at both listeners
Wrong name and wrong upstream identity are rejected
New root is loaded before new leaves are served
Fresh handshakes show the new key and leaf on every Pod
Old long-lived sessions follow the declared drain rule
Same-CA unrelated mTLS identity is denied by policy
Revoked test identity is rejected where checks are required
Exposed key has been replaced, not reused

Inject issuance and revocation faults

Use a test issuer path. Deny the challenge writer's DNS permission, observe the issuance error and retry behavior, then restore it and verify a single intended certificate is issued. Revoke a test client identity and measure when each gateway class rejects it; if a class does not check revocation, record that limitation and test the alternative short-lived or policy-denial control. Make the status service unavailable and observe the configured fail behavior. Track leaf and intermediate expiry independently.

Contain a synthetic key leak

Place only a disposable key in a restricted test artifact, then treat it as exposed. Generate a fresh key, deploy its certificate through a canary and full rotation, inspect old listener fingerprints and old session counts, and remove the test artifact. If the issuer key is the one exposed, use a trust-root replacement plan instead of leaf-only revocation. Report time to new identity, client failures, status-data age, and paths that still accept the old credential. Label any untested client or production gateway behavior unverified.

Common Mistakes

Do not copy a server private key into a client trust bundle.
Do not assume a Secret update proves the process serves a new certificate.
Do not present a revocation record as proof every client checks it.

Connected lessons

devops

project