Worker Leases, Retries, and Duplicate Effects

Working case

The report worker for job 84 stores an artifact and sends a completion message. It crashes after the artifact write but before acknowledging the queue message. A second worker receives the same job. If it creates a new artifact and sends another message, reviewer 47 sees two reports for one request. Instead, the artifact has a stable key derived from job ID and output revision, and the completion event uses an effect ledger. The second worker verifies the stored artifact and records the job as complete without repeating the external send.

Implementation boundary

function shouldApplyReportEffect(job, appliedEffects) {
  const effectId = `${job.id}:${job.outputRevision}`;
  return !appliedEffects.has(effectId);
}
console.log(shouldApplyReportEffect({ id: 84, outputRevision: 29 }, new Set(['84:29'])));
// Output: false

Use a worker state transition guarded by job version or compare-and-swap. Store the effect identity before or with the effect where possible; for an external provider, pass a supported idempotency key and keep an outcome ledger. Keep the queue lease long enough for normal work or extend it while work is healthy, but do not mistake lease tuning for deduplication. A retry policy classifies transport failures, temporary dependencies, validation faults, and authorization changes. Before each private data read, recheck job ownership and current policy. Acknowledge only after the durable result and job state are committed. Record a retry count and next attempt time without embedding private payload in logs.

Cost and boundaries

Each attempt may consume O(report bytes) processing and provider calls, so repeated full work can be expensive. A ledger lookup adds an indexed read per effect and storage proportional to retained effects. Lease extensions add broker calls but can avoid premature duplicate starts during long work. More concurrency increases throughput until database, provider, or CPU limits dominate. Bound workers per tenant and provider; retry storms after an outage should use delay and jitter rather than all workers waking together. Measure duplicate deliveries and duplicate effects separately.

Failure trace

Crash after storing the artifact but before queue acknowledgement. Redeliver the message and verify there is one final artifact and one completion notice. Expire a lease while the first worker still runs; a second worker may start, but only one guarded state transition can publish completion. Make the external provider time out after accepting a send, then retry using the same provider idempotency key where supported. Revoke the reviewer during a long job and confirm that result release follows current authorization. Inject a permanent malformed payload and prevent endless retries.

Verification

Redelivery does not duplicate the artifact or notice.
A lease expiry cannot bypass the effect ledger.
Retry classes separate transient from permanent failures.

Practice drill

Run job 84 twice with two worker identities. Pause the first after artifact creation and let the second lease expire path execute. Compare artifact IDs and effect-ledger rows after both resume. Add a provider timeout and a database deadlock as separate failure classes. Record which is safe to retry and which requires operator review. Test a burst of 83 jobs after an outage while observing downstream capacity.

Decision note

A lease limits simultaneous work temporarily; stable effect identity prevents duplicate business outcomes.

Common Mistakes

Calling a queue lease an exactly-once guarantee.
Acknowledging before durable effect state commits.
Retrying a provider call with a new effect identity.

Connected practice

Build Project: permit report job recovery and review Web Development: jobs and abuse decisions quiz.