Progressive delivery: canary checks and rollback

Last updated: 5 Oct 20266 min read

tutorial

Progressive delivery changes exposure in stages while observing a defined health signal. A canary is useful when the new version can run alongside the old one and the platform can route or select a subset of traffic. A rollback returns code or routing to a prior version; it does not undo an irreversible database write or restore deleted data.

Operational decision

A notifications API sends a small slice of requests to a new digest, then compares error rate and latency with the old version. Set the decision window before deployment, include enough requests to make the signal meaningful, and halt when the threshold is crossed. Avoid a canary that receives only low-risk traffic while the production change mostly affects high-risk paths. Track the new digest in each log and metric label without using unbounded request IDs as metric labels. The YAML sketch is a release policy document, not a controller-specific API. Automate pause and rollback only when the signal and control path have been tested; otherwise keep an operator ready with a known safe digest and a short command sequence. Follow the canary with a full-rollout watch because new failures may appear only at scale.

service: notifications-api
newVersionTrafficPercent: 7
observationMinutes: 23
abortWhen:
  errorRatePercentAbove: 1.8
  p95LatencyMsAbove: 420
rollbackDigest: sha256:9a09c4f113a2

Cost and verification

A canary consumes extra capacity because old and new versions overlap. Low traffic can make a short window statistically weak; set a minimum event count and use representative requests. Metrics arrive with delay, so a controller should not promote solely because no samples have appeared. The sample values are illustrative and require calibration against this service's baseline and SLO. Rollback may restore request handling quickly, yet a schema or queue-format change can make older code incompatible.