Kafka log compaction: rebuild state without resurrecting deleted keys

Last updated: 2 Oct 20267 min read

tutorial

Kafka log compaction eventually removes superseded records with the same key while retaining a recent value for keys still present. It does not immediately rewrite every segment or guarantee that a consumer sees only one record per key. A record with a key and null value is a tombstone: it says the key was deleted. Tombstones are themselves eligible for removal after the configured delete-retention interval. A reader rebuilding a complete materialized view from the beginning must finish its scan within a compatible window, or an old value can be observed without its later deletion marker. The application still needs an explicit snapshot and restore contract.

Operational decision

A customer-preferences view stores events keyed by customer ID. Key 7284 receives region west, then region north, then a null-value tombstone when the account is removed. A fresh view should end without key 7284. The operator measures full-bootstrap duration using the slowest realistic reader, including pauses and throttling, then chooses tombstone retention with headroom beyond that duration. It also tests a new reader from offset zero while the cleaner runs. If rebuilding takes longer than the tombstone window, use a consistent external snapshot with a captured stream position or increase the window; changing compaction frequency alone does not solve the deletion race.

key=customer-7284, value={region: west}
key=customer-7284, value={region: north}
key=customer-7284, value=null  # tombstone
Expected rebuilt state: customer-7284 absent
Reader bootstrap duration must fit deletion-marker window

Cost and verification

Compaction spends broker CPU, disk I/O, and temporary disk space. Larger tombstone windows preserve deletion evidence longer but also consume storage. A compacted topic is unsuitable as an immutable audit log because prior values may disappear; retain a separate history where the business requires it. Confirm the cleanup policy, key serialization, tombstone retention, and reader bootstrap time together. A null payload without the correct key is not a deletion of that customer's state. Observe cleaner backlog and materialized-view checksums under load before depending on a compacted topic for recovery.

Common Mistakes

Do not treat compaction as immediate per-key deduplication.
Do not rebuild a state store without testing tombstone retention against scan duration.
Do not use a compacted topic as the sole immutable history of every prior value.

Connected lessons

Practice and check

devops

kafka

stream-operations