Secret rotation changes a credential at its issuer and in the system that consumes it. A new value in a secret store does not guarantee that running processes reload it. Environment variables are fixed for a running container, and connection pools may keep old sessions alive even after a new credential is available. The rotation plan needs an overlap window or staged identities when the target service supports them.
Make this comfortable
Secret rotation rollout: update the issuer, consumer, and active connections
Operational decision
A claims API authenticates to a document database. Create a second narrowly scoped database identity, publish its credential through the approved secret path, and roll a small group of API Pods so they establish new connections. Observe authentication failures and connection age. The checklist is an operational contract, not a command that exposes values. When every active instance uses the new identity and a synthetic claim succeeds, disable the old identity and verify a denied login with it. If the storage system supports only one password, schedule a short controlled transition and make application refresh behavior explicit. Keep secret values out of logs, build artifacts, and support tickets. Test a failed rotation in a disposable environment so the recovery order is known.
Document database rotation gate
Create: new least-privilege identity
Distribute: approved secret path; no build artifact
Adopt: roll or refresh claims API instances
Verify: new connections and claim transaction succeed
Revoke: disable old identity at database
Negative test: old credential deniedCost and verification
Two valid identities add a temporary access surface, but permit a safer handoff when old and new processes coexist. A short overlap reduces exposure while increasing the risk of breaking a slow-to-refresh consumer; inventory all consumers first. Each rotation also exercises secret-manager availability and database authentication capacity. Monitor denied logins after revocation to find forgotten clients. Do not assume a mounted file refresh causes the application library to reopen its connection pool.
Common Mistakes
- Do not update only the secret store while leaving the issuer unchanged.
- Do not revoke before every consumer has adopted the new identity.
- Do not retain the old credential indefinitely after cutover.
Connected lessons
- DevOps: delivery, infrastructure, and reliable operations
- Secrets and configuration across the delivery path
- Credential incident response: revoke access before rebuilding trust
- Graceful Pod shutdown: stop accepting work before exit
- Service account tokens: mount only when the workload needs Kubernetes API access
Practice and check
Advanced follow-up
- Configuration pairs: prevent mixed policy and credential generations
- External secret sync: treat freshness as a monitored runtime contract
TLS trust operations follow-up
- Private CA rotation: overlap trust before changing issuers
- Certificate reloads: distinguish file delivery from active TLS state
systemd operating follow-up
devops
operations