Project: keep receipt data through Kubernetes storage transitions

Last updated: 5 Oct 20269 min read

project

Use a disposable Kubernetes cluster with a CSI driver that supports the selected access and expansion features, plus a separate local-volume test node. Create synthetic receipt data and an independent restorable copy before each destructive experiment. Record the claim, PV, provider asset, node or zone, application generation, reclaim policy, and owner. A bound claim is only the start of the acceptance check; finish with an application read and a verified recovery path.

Move a single writer

Create a ReadWriteOncePod claim, start one ledger writer, and try to start a second on the same node. Verify the second cannot write concurrently. Fail the first Pod during a synthetic transaction, then measure the new writer's mount and transaction-recovery time. Provision a zonal claim with WaitForFirstConsumer and confirm the selected PV zone agrees with the Pod's placement rules. Repeat when one zone lacks capacity, and record Pending and scheduling events instead of labeling every delay a storage outage.

Receipt storage acceptance gates
Writer: no concurrent Pod write to one claim
Placement: PV zone matches scheduled consumer
Expansion: backing device and usable filesystem both grow
Reclaim: expected asset remains or is deleted after claim release
StatefulSet: scaled ordinal retains its intended claim
Mount: ownership policy keeps access correct and startup bounded
Block: application recognizes the expected device generation
Local node loss: rebuilt data passes count and query checks

Grow, remove, and restore

Increase a test PVC request and verify both PV capacity and free bytes inside the container, then write past the prior capacity boundary. Scale a StatefulSet down and back up under Retain policies; compare ordinal and data generation. Delete a disposable claim first under Retain and then under Delete, checking the real provider asset each time. Mount a large copied volume with matching and mismatched fsGroup ownership to measure both paths. Present a raw block claim to a test application that knows its format and reject an unknown generation before writes. Finally remove a local-volume node, rebuild the affected index from a checkpoint, and keep it out of traffic until query results agree.

Cost and verification

Report storage billed after Retain, expansion charge, mount delay, idle ordinal cost, and rebuild traffic after node loss. Include claim and PV identifiers in the evidence without exporting user data. If a driver or cluster version lacks one feature, mark that case unverified and use a compatible isolated environment; a YAML parse is not a storage test.

Common Mistakes

Do not call a larger claim a successful expansion until the application can use the space.
Do not infer asset deletion from the PVC object alone.
Do not treat local storage as a movable replica.

Connected lessons

devops

project