Registry restore: reconcile aliases, approvals and serving pointers

Last updated: 6 Oct 20265 min read

tutorial

Recover registry state without promoting an unapproved version or discarding a legitimate post-backup change.

Freeze deployment while state is uncertain

After a registry database failure, serving replicas may still have a model loaded. Do not restart them blindly against an empty or stale registry. Pin current served digests from endpoint telemetry and stop automatic promotion while the restore runs. Restore the database and artifact objects into an isolated environment, then compare its active alias with the deployment pointer and append-only promotion records. Normal promotion gates remain in force during recovery.

Classify divergent states

There are several distinct cases: the restored alias points to an artifact absent from backup, serving runs a newer approved digest, serving runs a digest with no surviving approval, or the restored alias is older but safe. Each needs a disposition. Preserve the prior pointer and evidence; do not choose by timestamp alone because clocks and replication lag can disagree. Attestation is the authority for a valid promotion, while the backup manifest proves the bytes exist.

Rebuild a safe pointer

For each candidate digest, require available verified bytes, a compatible runtime, approval for the intended environment and a passing smoke contract. If any condition is missing, keep the last known-good serving digest and hold new deployments. Replay post-backup promotion events only after de-duplicating event IDs and validating expected previous pointers. A successful event can be reapplied once; a conflicting one must be reviewed instead of forced. Environment smoke tests cover the live runtime.

Reopen with an audit trail

Publish the recovered registry revision, artifact verification results, serving digest per cohort, lost or replayed promotion events and remaining exceptions. Confirm that the UI alias and actual endpoint agree before removing the freeze. Run a rollback test to the previous approved digest. The project includes an approved promotion after the backup boundary and a newer unapproved version that must stay dark.

Implementation

def restored_pointer(candidate, artifact_digests, approvals, runtime_ok):
    if candidate["digest"] not in artifact_digests:
        return "hold:missing-artifact"
    if candidate["digest"] not in approvals:
        return "hold:unapproved"
    if not runtime_ok.get(candidate["digest"], False):
        return "hold:incompatible-runtime"
    return "restore:" + candidate["digest"]

candidate = {"digest": "receipt-r47"}
assert restored_pointer(candidate, {"receipt-r47"}, {"receipt-r47"},
                        {"receipt-r47": True}) == "restore:receipt-r47"
assert restored_pointer({"digest": "receipt-r48"}, {"receipt-r48"},
                        {"receipt-r47"}, {"receipt-r48": True}) == "hold:unapproved"

Performance and operating cost

The gate is O(1) expected time and space for indexed evidence sets. A full reconciliation scans promotion events and affected deployments, O(p + d) for p events and d deployments. Freezing promotion delays releases, but is cheaper than silently reassigning production to a version whose approval or bytes did not survive.

Common Mistakes

Restarting all serving replicas before checking their loaded digests.
Trusting the restored alias as the source of truth by itself.
Replaying promotion records without validating the previous pointer.
Reopening deployment while the registry UI and serving endpoints disagree.