Model registry backups: keep metadata and artifact bytes consistent

Last updated: 6 Oct 20265 min read

tutorial

A registry restore fails if version records survive but the referenced model bytes or promotion decisions do not.

Treat the registry as more than a table

A production model registry commonly separates version metadata from large artifact objects. A database backup can restore a model version whose URI points to deleted bytes; an artifact copy without its approval and alias history cannot explain why a model served. Inventory the backend store, artifact store, immutable image store and promotion ledger. Define recovery point and recovery time objectives for each. Run lineage connects training to artifacts; promotion attestations connect artifacts to serving decisions.

Make a coherent recovery set

Capture a metadata checkpoint with a known time or transaction boundary, then ensure every artifact reachable from the retained registry versions exists in the backup set. Object copies may lag the database snapshot, and mutable aliases may move during copying. Prefer immutable version and digest references in the manifest; record alias state only after its target is confirmed. Keep encryption keys and access policy recoverable through a separate controlled process. A backup that cannot decrypt or load a model is not a usable backup.

Verify by reading, not by counting files

Restore into an isolated environment, query the active and prior versions, fetch each referenced artifact, verify its digest and load it with the intended runtime. Check approval records and the previous rollback target. Rehearse a failed artifact fetch and a metadata row whose object has not arrived. Digest verification detects corruption; runtime inventory tests whether recovered bytes can still execute.

Set a realistic recovery boundary

A recent backup may still lose a promotion after its checkpoint. Decide whether the restore should hold all serving pointer changes, replay an append-only promotion log or keep the last known-good deployment pinned until reconciliation. Never infer the correct active model from the highest version number. Restore reconciliation makes the alias decision explicit, and the project tests a metadata backup newer than one artifact copy.

Implementation

def verify_registry_backup(versions, artifact_digests):
    missing = []
    mismatched = []
    for version in versions:
        actual = artifact_digests.get(version["artifact_uri"])
        if actual is None:
            missing.append(version["version_id"])
        elif actual != version["expected_digest"]:
            mismatched.append(version["version_id"])
    return {"missing": missing, "mismatched": mismatched,
            "restorable": not missing and not mismatched}

versions = [{"version_id": "risk-r47", "artifact_uri": "objects/r47",
             "expected_digest": "sha256:47"},
            {"version_id": "risk-r48", "artifact_uri": "objects/r48",
             "expected_digest": "sha256:48"}]
report = verify_registry_backup(versions, {"objects/r47": "sha256:47"})
assert report["missing"] == ["risk-r48"]
assert not report["restorable"]

Performance and operating cost

Verification is O(v) expected time and O(v) worst-case result space for v retained versions; hashing artifact bytes adds O(b) work for b total bytes. Replicated storage and restore drills add cost, but an untested backup offers no measured recovery time. Keep the retention set aligned with active, rollback and audit obligations instead of copying every abandoned experiment forever.

Common Mistakes

Backing up registry rows while artifact objects are copied on a different schedule.
Restoring an alias to the newest version without an approval record.
Counting stored files instead of loading and hashing the referenced model.
Forgetting the key or runtime required to use restored bytes.