Environment promotion: keep the tested artifact and contract together

Last updated: 6 Oct 20265 min read

tutorial

Promotion moves an immutable model package through environments while proving the serving image and input contract still match.

Promote bytes, not a retraining recipe

A candidate trained in development should not be retrained from a mutable table when it enters staging. Move or reference the same immutable artifact digest, feature-contract version and evaluation record. The target environment may have different secrets, endpoints and capacity, but model bytes and declared behavior stay identifiable. Artifact verification checks what was received; promotion evidence binds the decision to those bytes.

Test environment-specific dependencies

Staging can use a different feature store, network policy, identity or hardware class. Run smoke requests against each target environment with expected admission, fallback and output states. Check that the serving image can read only the intended package and that its feature lookup has the required permissions. A passing unit test cannot prove a production service account can reach the correct data. Record the environment identity and observed model digest in the smoke result.

Control mutable pointers

A registry alias is convenient, but resolving it at different times can select different versions. Resolve the candidate and rollback aliases to immutable digests at the start of a release, then compare them again before moving traffic. Store the old and new pointer values in the release event. If a concurrent release changes the pointer, halt rather than silently overwriting it. Canary rollout needs a known rollback target throughout its observation window.

Make rollback a tested transition

Promotion is incomplete until the old package can load under the current feature producer and serving image. Stage a request against the rollback digest, then rehearse pointer restoration. If the new feature schema removes an old field, a model-only rollback may fail. The CI release project forces this compatibility test before any simulated customer traffic moves.

Implementation

def promotion_transition(candidate, tested, current, rollback):
    if candidate["digest"] != tested["digest"]:
        return {"state": "hold", "reason": "untested-bytes"}
    if candidate["feature_contract"] != tested["feature_contract"]:
        return {"state": "hold", "reason": "contract-mismatch"}
    if not tested["target_smoke_passed"]:
        return {"state": "hold", "reason": "target-smoke"}
    if current["digest"] != rollback["digest"] or not rollback["loadable"]:
        return {"state": "hold", "reason": "rollback-not-ready"}
    return {"state": "ready", "from": current["digest"],
            "to": candidate["digest"]}

candidate = {"digest": "sha256:r9", "feature_contract": "receipt-v4"}
tested = {**candidate, "target_smoke_passed": True}
current = {"digest": "sha256:r8"}
rollback = {"digest": "sha256:r8", "loadable": True}
assert promotion_transition(candidate, tested, current, rollback)["state"] == "ready"
assert promotion_transition({**candidate, "digest": "sha256:other"},
                            tested, current, rollback)["state"] == "hold"

Performance and operating cost

Comparing a fixed release record is O(1) time and space. Staging environments, load tests and dual-capacity canaries cost resources. The example checks a snapshot of the pointer; a production registry needs an atomic compare-and-swap or equivalent guard against concurrent releases.

Common Mistakes

Retraining during environment promotion and calling it the same candidate.
Assuming staging permissions match production permissions.
Resolving a mutable alias after the test instead of pinning a digest.
Keeping a rollback pointer whose model cannot load current features.