Project: permit report job recovery

Last updated: 5 Oct 20268 min read

project

Build a report workflow for case 47 at revision 29. The browser sends one idempotency key and receives a durable job ID plus an authorized status path. It does not receive a fake finished report. A repeated request with the same tenant, requester, revision, and payload returns the same job; a key reused for changed content returns a conflict. Commit the job row with an outbox intent so a crash between database commit and queue send cannot lose the work. The worker may receive a message more than once. It checks a stable effect ledger before creating the artifact or sending a completion notice. An expired lease can start a second worker, but only the current guarded state transition publishes the result. The status resource names queued, running, succeeded, failed, canceled, or partially canceled as separate outcomes. A private result link appears only after success and is checked against current record permission on access. An old-schema payload that cannot run moves to quarantine after bounded attempts without blocking healthy jobs. Replay retains its job and effect identity, rechecks current authorization, and records operator approval. A nightly digest of report completions uses a tenant and time-window claim; overlap coalesces instead of sending two messages. Cancellation stops future work at checkpoints, while a notice already sent remains visible as an escaped effect. The operator can initiate a corrective notice with its own ID. The UI distinguishes accepted work from server-confirmed completion in every path.

Build contract

Map duplicate admissions to one job or a conflict.
Make queue redelivery and lease expiry safe for business effects.
Quarantine permanent failures with controlled replay.
Expose cancellation, schedule overlap, and already-published effects honestly.

Implementation checkpoint

function mayPublishJob(job, effectLedger) {
  const effectKey = `${job.id}:${job.resultRevision}`;
  return job.claimVersion === job.currentVersion && !job.cancelRequested && !effectLedger.has(effectKey);
}
console.log(mayPublishJob({ id: 84, resultRevision: 29, claimVersion: 5, currentVersion: 5, cancelRequested: false }, new Set(['84:29'])));
// Output: false

Cost and boundaries

An indexed idempotency record and effect ledger add reads and retention storage, but prevent duplicate report CPU, storage, and provider sends. Rendering is O(report bytes) plus database work. Polling status every second from 83 tabs can become unnecessary load, so pause hidden tabs and use a bounded interval. Quarantine and replay add operator work; monitor oldest item age and failure class. A tenant claim reduces overlap while potentially delaying a large tenant. Limit catch-up windows after an outage rather than releasing an unbounded backlog. Measure queue age, duplicate delivery, duplicate effect count, cancel latency, and report completion by tenant.

Failure drill

Crash after writing the job row but before queue publication; the outbox must relay it. Crash again after artifact write and before acknowledgement; redelivery must leave one artifact and one notice. Let a lease expire with two workers active, then verify only the current claim can publish. Submit an unsupported schema and show quarantine after the chosen attempts, while 46 valid jobs continue. Replay the item twice and require one effect. Revoke the requester before result access. Cancel while rendering, then after a notice is sent; report the distinct outcomes. Pause the scheduler for two days and verify bounded catch-up rather than a burst of every missed tick.

Acceptance checks

One admission identity produces one report effect.
A job can move from accepted to a visible terminal failure.
Poison work does not block valid jobs and replay is audited.
Cancellation never claims to undo an escaped notice.

Common Mistakes

Acknowledging a message before durable state commits.
Giving a queued job a completed label.
Replaying quarantined work with a new effect identity.

web-tech

web-development