Project: stage and recover a stateful release

Last updated: 1 Oct 20269 min read

project

This project rehearses an upgrade to a stateful invoice index without treating Pod readiness as proof of data safety. Use a disposable cluster, a synthetic ledger, and a storage driver that supports snapshots. Record the old image digest, schema version, message position, and recovery objective before changing anything.

Stage the change

Create three StatefulSet replicas with retained claims and a tested backup. Hold the rollout partition at the highest ordinal, update only that replica, and run mixed-version reads. Add an optional normalized code field to a synthetic table, dual-write new rows, and backfill old rows in bounded ID ranges with a durable checkpoint. Introduce one malformed queue event and move it to the dead-letter destination after the configured attempt cap. Alert on oldest failed-event age.

Invoice release evidence
StatefulSet: partition held, ordinal and PVC recorded
Backfill: committed checkpoint and row-count sample
Queue: original event ID and unique effect key
Snapshot: restored into a separate claim
Stop gate: replication lag, user error, or duplicate effect
Decision: named operator with observed digest and schema

Fail and recover

Interrupt the backfill after one committed batch and restart it. Confirm that the checkpoint resumes without duplicate updates. Fix the malformed event, replay only its original ID through the normal consumer, and verify one ledger effect. Restore the snapshot to a new PVC and run a compatible database query; do not overwrite the only working volume. Advance the StatefulSet partition only when old and new replicas remain healthy under the same traffic mix. If the new version changes its on-disk format, exercise the actual restore path rather than claiming an image rollback is enough.

Cost and verification

The test uses extra storage, temporary compute, queue retention, and operator review time. Capture actual batch duration, replication lag, restored transaction position, and replay outcome. A green controller status is not proof of correct invoices. Clean up disposable claims only after the recovery evidence has been retained under the agreed policy.

Common Mistakes

Do not advance a partition because a single Pod is merely Ready.
Do not advance the backfill checkpoint before its transaction commits.
Do not replay with a fresh business key.

Connected lessons

devops

project