Project: prove configuration and identity delivery during a release

Last updated: 1 Oct 20269 min read

project

Use a disposable cluster and synthetic credentials. Separate the test namespaces for application, settlement signer, and CI deployer. Prepare an isolated encrypted backup and a restore cluster before experimenting with key retirement. Record the configuration generation and credential identifier, never their sensitive values. The acceptance result is a successful user-path request with the expected active generation, not the mere existence of Kubernetes objects.

Exercise propagation and startup

Update a mounted ConfigMap and measure API-to-file and file-to-process delay on every replica. Repeat through subPath and an environment variable, then document why those paths remain stale. Create a new immutable generation and roll it to a canary while preserving the prior object. Try a missing required object, missing key, and invalid policy; confirm no failing Pod receives traffic. Deliberately publish a policy and credential with mismatched generations and verify that startup validation rejects the pair. Correct both references and prove one authenticated synthetic transaction before promotion.

Configuration delivery acceptance gates
Projection: every replica reports file and active generation
Startup: missing or malformed required values block readiness
Pairing: policy and credential generation agree
External sync: active credential age stays inside the limit
Access: CI identity cannot create a Secret-reading workload
Encryption: retained backup restores with required decrypt key
Identity: broker accepts a refreshed token and rejects wrong audience

Exercise loss and recovery

Block the external issuer, rotate its synthetic credential, and time the lag through controller, mount, process, and successful request. Impersonate the CI deployer and attempt both direct Secret reads and workload creation in the signer namespace. Rotate an isolated cluster's API-data encryption key, rewrite test Secrets, and restore an older encrypted backup while its decrypt key is still available. Keep one projected-token Pod alive beyond its first token's expiry; verify the client reopens the file and that the broker rejects a different audience. For each fault, capture events and metrics without printing Secret or token bytes.

Cost and verification

Report mixed-generation duration, restart capacity, issuer request rate, KMS latency, object rewrite load, token refresh errors, and rollback time. Mark any drill unverified when the cluster or integration does not support it. Keep the old configuration object and decrypt key until the explicit rollback and backup windows close. A YAML parse and an RBAC can-i response are useful checks, but neither proves that the complete runtime path behaved correctly.

Common Mistakes

Do not equate a changed file with a changed process configuration.
Do not test only direct Secret get while ignoring workload creation.
Do not retire a key before testing restoration of an older backup.

Connected lessons

devops

project