Project: recover a versioned object archive

Last updated: 5 Oct 20269 min read

project

Use disposable object buckets, synthetic claims records, temporary credentials, and a queue-backed indexer. Define the recovery point and recovery-time objectives before injecting faults. Create three versions of a claim manifest with different record counts, then put a delete marker above them. Retain an external digest for each version. The exercise is accepted only when the correct version is readable through the application and its digest matches the signed manifest.

Recover the exact version

List versions and markers for one key. Record each version identifier, write time, digest, and claim ledger generation. Remove only the marker in one rehearsal; in another, copy a reviewed historical version forward and compare the resulting current version with the ledger. Apply a narrow lifecycle rule in a separate test prefix and show current, noncurrent, archive, and final expiry states. Record the earliest irreversible deletion point. If an archive tier delays retrieval, time the restore and compare it with the recovery objective.

Inject transfer and delivery faults

Start a multipart upload, send several parts, then kill the producer. Inspect incomplete uploads and bytes; allow the reviewed abort rule to clean up the abandoned identifier while a separate slow legitimate upload completes. Copy one completed object to a recovery bucket and compare exact versions, length, application digest, decryption, and parser result. Deliver the same object-created event twice and then reverse the order of two version events. The indexer must publish one active pointer for the newest approved generation and retain an auditable processed marker for each attempted version. Reconcile a deliberately suppressed event with an inventory or bounded live listing.

Archive recovery acceptance
Selected version matches claims ledger and digest
Current read returns the reviewed manifest
Abandoned parts are reclaimed after the retry window
Copied version decrypts and parses in the recovery account
Duplicate and reversed events do not regress the active index
Expired signed URL is denied and reissue checks authorization
Protected old version survives deletion attempt and remains decryptable
Inventory mismatch is sampled live and reviewed before action

Test protection and access

Issue a short-lived read URL for one object version, exercise it before and after expiry, and verify that no signed query string appears in logs. Repeat after signer credential expiry. Protect a separate version with retention and legal hold in the disposable account, test permanent deletion with an ordinary role, and observe whether a simple delete adds a marker. Restore the protected version with the recovery role and the required encryption key. Do not use a production hold or production customer document. Compare a scheduled inventory report with an application manifest, preserving the report generation time and classifying recently written objects as pending until a live check resolves them.

Cost and verification

Report recovery time, oldest verified recovery point, incomplete-part bytes, copy and readback bytes, inventory lag, event duplicates, missed-event age, signed-URL expiry failures, and key-denied restores. State which results came from the disposable provider environment and which were only reasoned from configuration. Include cleanup evidence for every bucket, queue, key, temporary role, and held test version. A successful API call alone does not satisfy the drill when the application still reads the wrong claim generation.

Common Mistakes

Do not erase a version while trying to clear a marker.
Do not use ETag equality as the only integrity check.
Do not bulk-delete inventory outliers without a live sample.

Connected lessons

devops

project