Model retirement: find consumers before removing a version

Last updated: 7 Oct 20265 min read

tutorial

A model version can be removed only after serving, batch, rollback, audit and replay consumers have explicit replacements or retention decisions.

Inventory live and latent consumers

The current endpoint is only one consumer. Nightly scoring may pin an old digest, a canary may still receive traffic, and incident rollback may depend on an earlier package. Historical replay, audit investigations and saved evaluation reports may also reference the artifact. List every alias, deployment, scheduled job, workflow and retention obligation by immutable digest. Reversible promotion assumes the old artifact remains loadable during the observation window.

Separate stop-serving from deletion

First stop new traffic and batch jobs from selecting the old version, then verify no hidden consumer resolves its alias. Mark it retired for new work but retain the package and manifest until rollback and evidence windows close. Deletion may be irreversible in a registry, and a renamed pointer can still be used by a worker that cached the old digest. Check actual invocation logs and scheduled definitions rather than assuming an empty dashboard means no dependency.

Preserve enough evidence to explain past decisions

A historical customer decision may need its model digest, feature contract, training manifest and decision record even if the raw training data reaches its retention limit. Retention rules differ for model packages, personal data and aggregated metrics. Keep the minimal lawful evidence and delete data under the applicable policy; an audit requirement is not a reason to keep unrestricted raw payloads forever. Inference privacy and lineage define what each record contributes.

Test a no-consumer state

Disable the old alias in staging, run endpoint and batch smoke tests, verify rollback now points to an approved successor and scan scheduled jobs for pinned digests. Require a grace period with zero observed use and an owner sign-off before deleting a package. The retirement project includes a hidden nightly batch consumer so the first deletion attempt must hold.

Implementation

def retirement_gate(model_digest, consumers, rollback_digests,
                    retention_hold):
    active = sorted(name for name, digest in consumers.items()
                    if digest == model_digest)
    if active:
        return {"state": "hold", "reason": "active-consumers", "names": active}
    if model_digest in rollback_digests:
        return {"state": "hold", "reason": "rollback-target"}
    if retention_hold:
        return {"state": "retain", "reason": "evidence-window"}
    return {"state": "eligible-for-review"}

jobs = {"online": "sha256:r8", "nightly": "sha256:r7"}
assert retirement_gate("sha256:r7", jobs, set(), False)["state"] == "hold"
assert retirement_gate("sha256:r7", {"online": "sha256:r8"},
                       set(), True)["state"] == "retain"

Performance and operating cost

Scanning c declared consumers takes O(c) time and O(a log a) for sorting a active names, with O(a) space. The hard part is inventory completeness: a stale configuration outside the registry may still load the artifact. Treat this function as one gate in an observed-use and retention review, not as deletion authorization.

Common Mistakes

Deleting a version because the main endpoint no longer uses it.
Removing an artifact still needed for rollback.
Keeping raw personal data indefinitely because the model package has an audit hold.
Assuming a registry alias inventory includes every pinned batch job.