Model promotion: require evidence before changing the serving pointer

Last updated: 7 Oct 20265 min read

tutorial

A model registry separates immutable candidate versions from the mutable pointer that traffic uses, with checks before promotion.

Register an immutable candidate

Store artifact digest, input and feature schemas, class map, training manifest, evaluation result and owner under a version. A human-readable alias can identify the active version, but the alias must resolve to one immutable artifact at deployment time. Run lineage proves the candidate’s origin.

Make gates executable

Require complete metadata, schema compatibility, a known-good smoke set and measured latency within a declared budget. Compare candidate quality against the incumbent on the same evaluation population; allow explicit exceptions only with recorded rationale. Do not substitute a training score for an independent evaluation. Check whether the candidate artifact uses a loading format that can execute code, and accept only trusted artifacts under a controlled environment.

Promote as a state change

After all gates pass, update a version pointer or alias. Record who promoted it, when, why and the previous pointer. A rollback should point to a still-available prior artifact and matching preprocessing. Canary rollout can test behavior before the full pointer change.

Drill rejection and rollback

Try a candidate with missing class mapping, a mismatched preprocessing hash and a failing known-good request. Each must be rejected. Then promote a valid candidate and roll back; verify request outputs return to the previous version. A registry page alone does not prove the serving process loaded the intended artifact.

Implementation

REQUIRED_FIELDS = {"artifact_sha256", "data_snapshot", "feature_schema",
                   "class_names", "evaluation_manifest", "preprocess_version"}

def approve_candidate(manifest, smoke_passed, latency_p95_ms, budget_ms):
    missing = REQUIRED_FIELDS - manifest.keys()
    if missing or not smoke_passed or latency_p95_ms > budget_ms:
        raise ValueError(f"candidate failed promotion gate: {sorted(missing)}")
    return manifest["artifact_sha256"]

Performance and operating cost

Metadata checks are O(F) for F required fields; loading and smoke-testing the candidate dominate time and memory. Retaining one prior version costs storage but makes rollback practical.

Common Mistakes

Do not change a serving alias before validation finishes.
Do not lose the prior artifact when promoting a candidate.
Do not load untrusted serialized model files.