Replica version erasure and restore guards

Last updated: 6 Oct 20265 min read

tutorial

Replica erasure must account for every object version and restore path, because hiding a current key does not remove historical bytes or prevent an old backup from republishing them.

Separate logical from physical deletion

A delete marker can make the current object unreadable while older versions still occupy storage. A replication rule may copy the marker, omit it, or leave version-specific deletion to the destination operator. Do not infer destination erasure from a successful source delete. Enumerate versions by object identity and region, including retained snapshots, index generations and dead-letter copies. The deletion ledger should track each surface independently.

Freeze resurrection paths

A backup restored after an erasure can bring back a pre-deletion customer row. Persist a durable deletion ledger outside the backup being restored, with entity key and monotonically ordered deletion position. Before any restored generation becomes queryable, replay the ledger through the restored data and serving index. A lagging replica must reject an older upsert after the deletion position.

Respect retention holds explicitly

Immutable retention or a legal hold may prevent immediate physical removal. Record the hold owner, scope, expiry condition and access restrictions; do not mark the bytes erased while the hold remains. Once the hold is released, expire the applicable object versions and snapshots, then verify that the cleanup job completed. An audit claim must distinguish inaccessible, logically deleted and physically removed states.

Prove completion at every destination

For each entity and copy, collect version listing, current-read test, snapshot reference check and cleanup result. A current-read 404 is insufficient if a version-specific read still succeeds. Include cross-account replicas, old disaster-recovery regions and staged backfills. Report partial failure as pending, retry with bounded backoff and prevent publication of a restored copy until checks pass.

Test an adversarial restore

Delete customer 47 at source position 73, then restore a snapshot ending at 71. The replay gate must remove the customer before opening the restored table. Inject a delayed update at 72 and verify it cannot recreate the row. Check the current table, index and raw version inventory, then record separate evidence for each. Placement policy still applies to the restore location.

Implementation

restored_rows = {"customer-47": {"position": 71, "cents": 2300}}
deletions = {"customer-47": 73}

def guard_restore(rows, deletion_positions):
    safe = dict(rows)
    for customer_id, delete_position in deletion_positions.items():
        if customer_id in safe and safe[customer_id]["position"] <= delete_position:
            safe.pop(customer_id)
    return safe

assert guard_restore(restored_rows, deletions) == {}
late_update = {"customer-47": {"position": 72, "cents": 2400}}
assert guard_restore(late_update, deletions) == {}

Performance and operating cost

The reference guard is O(K) time for K deletion keys and O(N) copied row state. A production restore can avoid scanning every record by indexing deletion keys and affected partitions, but must still verify all replicas and historical versions. Version inventory, retention holds and cross-region cleanup make the full erasure cost proportional to the number of copies and retained generations, not just current rows.

Common Mistakes

Do not treat a delete marker as proof that all versions are gone.
Do not restore a backup before replaying later deletions.
Do not claim physical erasure while an immutable hold still retains bytes.