Serverless event idempotency: commit the effect and receipt together

Last updated: 5 Oct 20266 min read

tutorial

A queue or event source may redeliver an event after a worker fails, a visibility lease expires, or acknowledgement is lost. In-memory deduplication cannot cover a new invocation. A durable idempotency key must identify the business operation, not merely the transport attempt; the effect and its receipt should be committed in one transaction when they share a database. If the effect spans an external service, use a stable request key there and reconcile an uncertain response. A receipt written before the effect can lose work; a receipt written afterward in a separate transaction can repeat it.

Operational decision

A settlement function receives the same payout event twice after its first acknowledgement is lost. It inserts a receipt keyed by settlement ID and writes the ledger movement inside the same database transaction. The second invocation sees the existing receipt and returns the stored outcome. A new event with the same key but a different amount is a conflict for investigation, not a silent duplicate. If the processor sends a bank instruction, pass a stable idempotency key to the bank and keep an uncertain state until the bank's status API confirms the result. Exercise the crash immediately before commit and immediately after commit; both replays must leave one movement.

BEGIN;
WITH accepted AS (
  INSERT INTO settlement_receipts (settlement_id, payload_hash, result_status)
  VALUES ('stl-8472', 'payload-digest-47', 'applied')
  ON CONFLICT (settlement_id) DO NOTHING
  RETURNING settlement_id
)
INSERT INTO ledger_movements (settlement_id, account_id, amount_cents)
SELECT settlement_id, 'reserve-claims', 4731 FROM accepted;
DO $$
BEGIN
  IF EXISTS (
    SELECT 1 FROM settlement_receipts
    WHERE settlement_id = 'stl-8472'
      AND payload_hash <> 'payload-digest-47'
  ) THEN
    RAISE EXCEPTION 'settlement payload conflict';
  END IF;
END $$;
COMMIT;

Cost and verification

A unique-key lookup is typically O(log N) with a B-tree index; the receipt table grows with processed operations and needs retention aligned to the replay horizon. The SQL assumes settlement_receipts has a unique settlement_id and that ledger_movements also forbids duplicate settlement IDs. Only an inserted receipt can feed the movement in this transaction; a changed payload raises an error and rolls back. In application code, bind the validated digest and amount as parameters instead of interpolating event text. Test concurrent duplicates and record receipt conflicts, uncertain external outcomes, and replay age.

Common Mistakes

Do not deduplicate only in process memory.
Do not mark an event complete before its effect commits.
Do not treat the transport message ID as the business identity without checking producer behavior.

Connected lessons

Practice and check

devops

serverless