A CustomResourceDefinition can serve more than one API version while storing objects in one chosen version at a time. Changing the storage version does not rewrite old objects automatically. A read served as the new version may be converted on demand while the underlying object remains in the old stored version. Removing the old version too early can break reads, watches, or controller upgrades.
Make this comfortable
CRD storage migration: retire an API version only after stored objects move
Operational decision
A billing operator moves its InvoiceRun resource from v1beta1 to v1. Keep both versions served while clients change. Test conversion in both directions with real old objects, unknown fields, defaults, and status updates; conversion must preserve the business meaning of each object. Deploy and observe a highly available conversion webhook before setting v1 as storage. Migrate stored objects with a supported migration process, record progress and failures, and verify the CRD's storedVersions no longer lists v1beta1 before dropping it from the specification. The read-only sample reports the served and stored declarations; it does not migrate data. Exercise a watch and a rollback client against a copy of the data. If the conversion webhook is unavailable, even reads of old stored objects requested in the new version can fail. Keep the old path until the rollback window and every cluster's migration evidence are complete.
kubectl get crd invoiceruns.billing.aitrove.test -o jsonpath='{.spec.versions[*].name}{"\n"}{.status.storedVersions}{"\n"}'
kubectl get invoiceruns.billing.aitrove.test -A --request-timeout=10sCost and verification
Serving two versions and running conversion adds API latency, webhook capacity, and upgrade coordination. A bulk rewrite adds etcd writes and can contend with controllers, so throttle it and checkpoint progress. Removing an old version reduces maintenance only after stored objects and clients have moved. Do not patch storedVersions just to make a rollout pass; that status must reflect completed storage migration.
Common Mistakes
- Do not infer stored data was rewritten because a new-version GET succeeds.
- Do not remove old conversion support while storedVersions still contains the old version.
- Do not treat an untested webhook as a safe rollback path.
Connected lessons
- DevOps: delivery, infrastructure, and reliable operations
- GitOps rollout order: install API definitions before dependent objects
- API compatibility windows: release consumers and producers safely
- Admission webhook outage: choose a deliberate failure path
- Database backfills: checkpoint progress without racing live writes
Practice and check
devops
operations