Queue consumers: acknowledgement, idempotency, and backlog

Last updated: 5 Oct 20266 min read

tutorial

A queue decouples producers from consumers, but many delivery systems can redeliver a message after a crash, lost acknowledgement, or expired visibility interval. The consumer therefore needs an idempotent business operation, not an assumption that every message arrives once. Acknowledgement should follow durable processing; acknowledging first can lose work if the process stops before the side effect commits.

Operational decision

A refund-notification worker receives a refund event with a stable event identifier. In one transaction, insert an idempotency record and the intended outbox notification; if the event identifier already exists, return success without creating another notification. Commit before acknowledging the queue message. The SQL sketch shows the deduplication gate, not the entire worker protocol. Set the consumer's in-flight limit according to memory and downstream capacity. Monitor oldest-message age, retries, dead-letter volume, and completed business actions; CPU alone may stay low while backlog grows. If processing fails permanently, move the message to an inspected dead-letter path with a replay procedure. A replay must preserve the same event identifier so the deduplication contract still applies.

BEGIN;
INSERT INTO processed_refund_events (event_id, processed_at)
VALUES (:event_id, CURRENT_TIMESTAMP)
ON CONFLICT (event_id) DO NOTHING;
-- Only when the insert affected one row: write the notification outbox row.
COMMIT;
-- Acknowledge the queue message after the transaction commits.

Cost and verification

The idempotency table and outbox consume storage; retain records for at least the period in which messages may be replayed, and align cleanup with the queue's retention policy. A small prefetch protects downstream systems but can lower throughput, while a large prefetch increases unacknowledged work after a crash. The SQL assumes a database with ON CONFLICT support and a unique event_id constraint. Duplicate deliveries are normal operational input, not proof that the broker is broken.