RabbitMQ delivery recovery project: confirms, quorum, and queue cutover

Last updated: 5 Oct 202612 min read

project

Use an isolated three-node RabbitMQ cluster and synthetic report requests. Declare a quorum work queue, a quarantine queue, and their exchange bindings. Give each request a stable business ID and keep an external expected-ID ledger. Enable publisher confirms, mandatory returns, and manual consumer acknowledgments. The exercise passes only when accepted publish IDs match the durable work or completed-effect ledger after each fault; a green connection indicator alone is insufficient.

Prove publish and consume boundaries

Remove a binding and publish 73 synthetic requests; verify mandatory returns are handled and none are falsely marked delivered. Restore the binding, publish again, and kill the producer before its last confirm arrives. Retry ambiguous IDs and confirm that the consumer effect ledger rejects duplicate report generation. Set bounded prefetch near actual worker slots. Crash a worker after it records a completed report but before it acknowledges, then show that redelivery does not create a second report. Capture ready and unacknowledged depth separately.

Acceptance ledger
Requested report IDs: 73
Unroutable test: 73 returned, 0 marked delivered
After binding repair: accepted IDs match queue or completed ledger
Producer lost-confirm retry: no duplicate report effect
Worker crash after effect: redelivery guarded by business ID
Two quorum members lost: queue unavailable until majority restored

Inject broker and dead-letter faults

Stop one queue member, verify progress, then stop a second and record the failed operation. Restore the majority and reconcile confirmed publish IDs. Reject a poison event through a bounded delivery path. Remove the dead-letter target binding and measure source retention under the selected transfer strategy; repair the binding and inspect the quarantine receipt. Publish messages with a short TTL behind a paused consumer and compare expiry counters with actual disk use. Ensure the worker rejects a business deadline that has passed even when delivery occurs near expiry.

Cut over queue type

Declare a second queue with the intended type and bindings. Fence the old publisher route at a recorded event ID, drain old deliveries, and start the new route. Reconcile source receipts, both queue depths, unacknowledged counts, and completed request IDs. Test rollback while the new queue still contains work. Preserve the commands, expected ledger, observed faults, and repair decisions; do not execute the exercise against production credentials.

Common Mistakes

Do not count confirms without checking unroutable returns.
Do not delete a queue with unacknowledged work.
Do not change dead-letter overflow without testing the transfer contract.

Connected lessons

devops

project