Snapshot-safe delete rewrites

Pin the starting snapshot

A maintenance worker reads snapshot 74 while ingestion continues. It must identify the exact data files, applicable deletes and sequence positions it intends to replace. An unpinned scan can mix two table states and produce a candidate that never corresponded to one valid snapshot. Optimistic commit rules decide whether the replacement still applies when another writer advances the table.

Preserve applicability

A position delete names a file and row location; moving rows to a new file changes that location. A rewrite must apply the old delete before writing surviving rows, then retire only metadata whose scope is fully replaced. An equality delete may also affect other live files. Do not discard it merely because one rewritten file no longer needs it. Retain or transform it according to the table format and sequence contract.

Validate two snapshots

Compare visible key sets, duplicate counts and monetary controls between the pinned source and candidate. Also query one older retained snapshot through its original metadata after the new commit. A maintenance job that produces the right current answer but garbage-collects files still referenced by an older snapshot breaks time travel and rollback. The manifest should name both snapshots and their file inventories.

Handle concurrent writes explicitly

If an ingestion commit adds a newer row or delete while compaction runs, either rebase the candidate after conflict validation or retry from the new snapshot. Never publish a replacement that silently drops the concurrent change. Bound retries and surface repeated conflicts; a hot partition might need a smaller maintenance slice or a quiet window rather than an infinite retry loop.

Separate commit from cleanup

A new snapshot can be committed before obsolete files are eligible for physical removal. Keep cleanup asynchronous and governed by replay horizons, legal holds and replica lag. Verify no retained snapshot references a file before removing it. A delete request may require an independent erasure workflow that can shorten the normal retention path only where policy permits.

Implementation

source_rows = {"order-47": 2375, "order-48": 6400, "order-49": -125}
deleted = {"order-48"}

def visible_rows(rows, removed):
    return {order_id: cents for order_id, cents in rows.items()
            if order_id not in removed}

candidate = visible_rows(source_rows, deleted)
assert candidate == {"order-47": 2375, "order-49": -125}
assert sum(candidate.values()) == 2250
assert source_rows["order-48"] == 6400  # retained snapshot still has its bytes

Performance and operating cost

Filtering N rows uses O(N) expected time and O(N) candidate space. Real snapshot validation also reads metadata and often the rewritten data, so its I/O is proportional to selected files. Retaining old files costs storage until expiry, but deleting them early can invalidate rollback. Conflict retries multiply rewrite cost and should be measured separately from successful maintenance.

Common Mistakes

• Do not remove an equality delete that still applies to another live file.
• Do not garbage-collect files referenced by retained snapshots.
• Do not ignore an ingestion commit that lands during a rewrite.

Read next

• Optimistic table commits and write conflicts
• Table snapshots and atomic publication
• Compaction, retention and the replay horizon
• Delete-file amplification and compaction
• Project: reduce delete overhead without breaking old snapshots

Continue the workflow: Lake file statistics and safe data skipping.