Serverless rollback: restore code traffic without assuming data rolled back

Last updated: 5 Oct 20266 min read

tutorial

Function releases often package code, assets, bindings, and configuration into a version, then route traffic to one or more versions. Redirecting traffic to an earlier version does not reverse database writes, queue acknowledgements, cache entries, or storage mutations. A safe rollout keeps both versions compatible with the active schema and event format for the rollback window. For asynchronous consumers, in-flight messages may be processed by both versions around a shift; record the version alongside each effect and avoid a destructive schema contraction until old workers have drained.

Operational decision

A receipt API deploys version B to a small cohort. B writes a new optional tax field while A remains active. The team observes a mismatch in tax rounding and routes new traffic back to A. Before declaring recovery, inspect receipts created by B, reconcile affected totals, confirm A can read them, and identify queued events still using B's payload format. Keep the schema additive until the replay horizon closes. A rollback drill should include a write made by B, an A read of that record, a delayed event, and a cache key created under B. Compare error rates by version rather than aggregating them into one fleet average.

Receipt release gate
A: current handler, reads optional tax field
B: candidate handler, writes optional tax field
Canary: cohort and version-specific errors
Rollback: route new calls to A
Reconcile: B-written records and delayed events
Contract: no schema removal before replay window closes

Cost and verification

Version routing changes are O(1) control-plane updates, but recovery cost scales with affected writes and queued events. Retaining two compatible paths adds temporary code and storage cost; removing one too early can turn a quick traffic rollback into data repair. Track version-specific failure rates, outstanding event age, and the count of records requiring reconciliation before ending the incident.

Common Mistakes

Do not call a code rollback a data rollback.
Do not contract the schema while old invocations can still run.
Do not aggregate away a small failed canary cohort.

Connected lessons

Practice and check

devops

serverless