In a versioned object store, a delete without a version identifier can create a delete marker that becomes the current version. A normal read of the key then appears missing, while earlier object versions remain addressable by version identifier. A delete that names a specific version is a different operation: it can permanently remove that version unless retention blocks it. Recovery starts by identifying the exact key, bucket, version history, and current marker; a blind re-upload can overwrite the business meaning of the key.
Make this comfortable
Object version recovery: distinguish a delete marker from lost bytes
Operational decision
A settlement service reads daily receipt manifests from a versioned archive. After an operator deletes a manifest key, freeze automation that could expire or overwrite it. List versions and markers for that exact key with a read-only role. Compare the candidate version's creation time, checksum, and manifest record count with the authoritative settlement ledger. In an isolated rehearsal, restore visibility by removing only the current delete marker or by copying the selected old version to a new current version, according to the approved recovery policy. Record both old and new version identifiers. If later writes happened after the marker, removing it alone may expose the wrong version, so check the complete ordering first. Restrict permanent version deletion to a separate role and require a retention decision before any cleanup. Test the reader path, not just the storage API: an application cache may still return a stale miss after the object is visible.
aws s3api list-object-versions --bucket receipt-archive-drill --prefix settlements/2026-09-27/manifest.json
aws s3api head-object --bucket receipt-archive-drill --key settlements/2026-09-27/manifest.json --version-id reviewed-version-idCost and verification
Listing V versions for one key takes O(V) review work and may require pagination; the actual restore is a small number of metadata or copy operations, but a copy can incur storage and request charges. Measure time to identify the authoritative version, time until the application reads it, and the age of the latest verified version. Versioning consumes storage for old bytes; pair recovery requirements with reviewed noncurrent-version retention rather than assuming old versions remain forever.
Common Mistakes
- Do not equate a current-key 404 with permanent loss of every version.
- Do not delete a version identifier while trying to remove only a marker.
- Do not restore solely by timestamp when the ledger or checksum contradicts it.
Connected lessons
- DevOps: delivery, infrastructure, and reliable operations
- Backups and disaster recovery: prove the restore path
- Immutable backup retention: protect recovery copies from deletion
- Object replication: verify the exact recovery object arrived
- Encryption key rotation: keep old data decryptable during recovery
Practice and check
devops
object-storage