Object events: survive duplicate delivery and stale notifications

Last updated: 5 Oct 20266 min read

tutorial

Object-storage notifications can be delivered more than once and may arrive out of order. A consumer that keys only on object name can process an older overwrite after a newer one, or publish duplicate downstream work for the same version. Record the object version or another immutable generation identifier and make the resulting side effect idempotent. Where a provider includes a per-key ordering token, compare it only in its documented scope; do not turn it into a global ordering guarantee.

Operational decision

A claims-index service receives object-created events through a queue. Its database stores one row per bucket, key, version, and processing generation, with a unique constraint on that tuple. On each event it checks the exact version, obtains a lease, validates checksum and schema, then commits the index pointer and processed marker in one transaction. A duplicate event sees the committed marker and exits. An old event may still produce a historical index if policy requires it, but it cannot replace the active pointer without a compare-and-set against the recorded current generation. Test duplicate delivery, reversed event order, missing object, a worker crash after indexing but before queue acknowledgement, and a poison file. Use an inventory or periodic listing to reconcile objects that never produced a usable event. Keep replay windows and dead-letter retention long enough for incident response, while avoiding customer data in event logs.

Claims index event identity
Idempotency key: bucket + object key + version ID + processor generation
Read: version-specific object and checksum
Commit: index pointer and processed marker atomically
Duplicate: detect committed marker and acknowledge
Reordered: older version cannot replace newer active pointer
Gap repair: periodic inventory reconciliation

Cost and verification

For E delivered events, processing is O(E) plus object-read cost; a keyed unique index gives expected logarithmic lookup in a B-tree and storage proportional to distinct processed versions. Deduplication does not prevent missed notifications, so reconciliation is still necessary. Measure duplicate rate, event-to-index lag, dead-letter age, and inventory-to-index gaps. A queue can smooth bursts, but unbounded replay can overload the indexer unless concurrency is capped.

Common Mistakes

Do not use object key alone as an idempotency key when overwrites are allowed.
Do not assume delivery order matches write order across all keys.
Do not remove periodic reconciliation merely because notifications are enabled.

Connected lessons

Practice and check

devops

object-storage