Private CA rotation: overlap trust before changing issuers

Operational decision

A warehouse scanning fleet authenticates to an internal inventory API. The platform team versions its trust bundle and rolls a temporary bundle containing both current and incoming roots to scanners, API clients, gateways, and the disaster-recovery site. Each client reports the loaded bundle generation; a configuration file update alone is not proof that a long-lived process refreshed its TLS context. The team then issues a new API leaf from the incoming CA and tests old and new clients against the same listener. Any scanner that cannot accept the new root is isolated and upgraded before the issuer switch. After the longest permitted client update window, verify no connection still relies on the old chain, remove the old root, and test that a certificate signed only by the retired root is rejected. Use different bundle names or generations so rollback is explicit. If the old CA key is compromised, the ordinary overlap schedule may be unsafe; invoke the compromise response instead and accept that some clients may need emergency updates. Track the expiry of roots and intermediates independently of leaf renewal because rotating a CA Secret may not automatically reissue every leaf.

Warehouse CA transition
Phase 1: distribute old plus new roots to approved clients
Gate: process reports loaded bundle generation
Phase 2: issue leaf from new chain and test both client cohorts
Phase 3: migrate remaining clients and recovery site
Phase 4: remove old root and reject old-only chain
Exception: suspected CA-key compromise uses emergency path

Cost and verification

Trust distribution costs O(C) updates for C client deployments, but mixed versions can persist for days if edge devices connect infrequently. A dual-root interval briefly trusts both issuers, increasing the set of acceptable certificates; bound its duration and issuance authority. Measure clients on each bundle generation, handshakes still using the old chain, root and intermediate expiry, and time to remove the retired anchor. A passing server probe does not prove every client has refreshed its trust store.

Common Mistakes

• Do not switch to the new issuer before clients trust its root.
• Do not mount a server key Secret into clients merely to obtain a CA certificate.
• Do not assume rotating a CA Secret automatically reissues all leaves.

Connected lessons

• DevOps: delivery, infrastructure, and reliable operations
• Secret rotation rollout: update the issuer, consumer, and active connections
• Encryption key rotation: keep old data decryptable during recovery
• Mesh identity: require encrypted peers and narrow service access
• Certificate renewal: verify the served certificate after issuance

Practice and check

• Project: rotate service trust and contain an exposed TLS key
• DevOps TLS trust and rotation decisions