Project: prove a serverless release survives duplicate events and rollback

Last updated: 5 Oct 202610 min read

project

Build a disposable receipt-processing service with an HTTP function, a queue consumer, a scheduled reconciliation job, and a database. The acceptance target is one settled receipt per business event under retries, bounded database pressure during a burst, and an earlier handler version that remains able to read records written during the canary. Use a local emulator or disposable provider account; do not run the faults against live financial data.

Define each boundary

Give the HTTP path a caller deadline and explicit time allocations for authentication, lookup, and response. Limit the function's concurrency from measured database headroom. Give each event a stable settlement ID, a stored payload hash, and one unique movement key. Give the scheduled job an interval identity, durable checkpoint, lease expiry, and fencing generation. Save immutable function versions and the active schema before the canary.

Receipt drill acceptance
Deadline: slow dependency returns a bounded result
Concurrency: database remains below its tested safe occupancy
Duplicate event: one ledger movement after two deliveries
Schedule: expired holder cannot commit after lease transfer
Canary: version-specific outcome is observable
Rollback: prior handler reads candidate-written records
Repair: uncertain external effect is reconciled before retry

Inject failure and recover

Delay the ledger call until the client budget expires and inspect CPU time separately from elapsed duration. Replay the same queue message after commit but before acknowledgement; verify one movement and one stored outcome. Raise concurrent HTTP calls until the configured ceiling rejects or queues work while the database remains stable. Stop a scheduled run after checkpoint 17, let another holder resume, and prove the old holder cannot commit. Deploy a candidate that writes an optional receipt field, route a small cohort to it, then send new traffic back to the prior version. Read candidate-written records through the old version and reconcile any changed totals.

Report evidence

Record duration, CPU, throttles, oldest event age, database connections, duplicate attempts, lease transitions, and version-specific error rates. State which failure was contained and which work remained pending. A passing HTTP response is insufficient if a queued settlement was duplicated or a scheduled interval silently skipped.

Common Mistakes

Do not count a trigger as a completed business operation.
Do not infer database safety from function success alone.
Do not assume redirecting traffic reverses writes.

Connected lessons

devops

project