RabbitMQ consumers: couple manual acknowledgments to a bounded prefetch window

Last updated: 5 Oct 20267 min read

tutorial

RabbitMQ manual acknowledgments separate delivery from completion. A message remains unacknowledged until the consumer sends an acknowledgment on the channel that received its delivery; a closed channel can cause those outstanding deliveries to be offered again. Prefetch limits how many unacknowledged deliveries are sent ahead of completed work. Without an effective limit, a fast broker can fill consumer memory even when processing is slow. Prefetch is not a concurrency setting by itself: the worker's actual task limit, downstream capacity, and message size must agree with it. Delivery tags are scoped to a channel, so an acknowledgment on another channel is invalid.

Operational decision

A document-indexing worker has eight execution slots, each using about 14 MiB of working memory. Start with prefetch near the execution-slot count and measure throughput, unacknowledged messages, resident memory, and downstream search pressure. After an index write and its unique document-event receipt commit, acknowledge on the original channel. If the write fails transiently, release the delivery through a bounded retry path rather than hot-looping immediate requeues. If a worker dies after indexing but before acknowledging, the same event can return; the unique receipt turns that delivery into a no-op. Verify this crash point explicitly in the fault drill.

Worker slots: 8
Per-task working set: about 14 MiB
Initial prefetch target: 8, then measure
Effect: index update + unique event receipt
Acknowledgment: original delivery channel, after effect commit
Crash after commit: redelivery rejected by receipt key

Cost and verification

A larger prefetch can improve throughput when processing is cheap and network latency dominates, but it also increases outstanding memory and redistributes work less fairly among consumers. A prefetch of zero can mean no limit under AMQP 0-9-1, so it should not be used to mean pause. Measure queue-ready, unacknowledged, consumer CPU, and processing age separately. A low ready count with a high unacknowledged count may indicate consumers holding work rather than finishing it. If completion takes longer than an acknowledgment timeout, increase capacity or redesign the work unit before simply extending the timeout.

Common Mistakes

Do not use automatic acknowledgment for work that must survive a consumer crash.
Do not acknowledge a delivery on a different channel.
Do not tune prefetch from ready-message count alone.

Connected lessons

Practice and check

devops

rabbitmq

queue-operations