Runtime patch canaries: security fix without model drift

Last updated: 7 Oct 20265 min read

tutorial

Patch a serving dependency with an explicit replay and host-class canary, then keep a safe rollback decision ready.

Classify the patch boundary

A loader, framework or execution-provider update can alter deserialization, graph optimization and numeric kernels even when the model digest is identical. Record the issue, affected version range, fixed build, exposure path and urgency. If the vulnerable code is unreachable in the configured serving route, document that evidence rather than assuming exposure from a package name alone. If it is reachable, contain the route while the patch is tested. The inventory identifies each deployment and owner.

Replay the old and new stacks

Hold model bytes, preprocessing and inputs constant while comparing the old and patched runtime. Test load failures, output schema, score distances, threshold crossings, p99 latency, memory and fallback behavior by hardware class. A security patch may fix one exploit while breaking a custom operator. Do not waive functional checks because the change is “only infrastructure.” Score parity and hardware qualification provide reusable gates.

Stage by exposure and host class

Deploy the patched image to a small compatible cohort, verify selected provider and actual model digest, then watch crashes, route rates and tail latency. A single global canary can miss the only host class that uses a vulnerable native library. Keep the rollout window short enough for the issue’s urgency, but long enough to collect exposed decisions. Coverage evidence prevents a quiet dashboard from being read as a pass.

Choose a safe recovery route

Rolling back to a vulnerable image may be unacceptable. Prepare an alternate known-safe image, disable the affected model route or move a cohort to a supported provider if parity and capacity allow. Record a decision tree before rollout. A rollback plan that merely says “restore previous version” is incomplete when the previous version contains the issue. The project exercises exactly that choice and keeps the corrected host class on a safe fallback until its patched provider passes replay.

Implementation

def patch_disposition(patch, canary):
    if not patch["fixed_build_verified"]:
        return "hold:unverified-fix"
    if canary["route_flips"] or not canary["provider_verified"]:
        return "fallback:safe-provider"
    if canary["p99_ms"] > canary["p99_budget_ms"]:
        return "fallback:safe-provider"
    return "expand:patched-image"

patch = {"fixed_build_verified": True}
canary = {"route_flips": 0, "provider_verified": True,
          "p99_ms": 71, "p99_budget_ms": 89}
assert patch_disposition(patch, canary) == "expand:patched-image"
assert patch_disposition(patch, {**canary, "route_flips": 2}) ==        "fallback:safe-provider"
assert patch_disposition({"fixed_build_verified": False}, canary) ==        "hold:unverified-fix"

Performance and operating cost

The disposition gate is O(1) time and space. Full patch qualification costs replay, hardware-cell benchmarks, a staged cohort and temporary duplicate images. Urgent fixes compress the observation window, increasing uncertainty; risk can be reduced by narrowing affected routes and using a tested safe provider rather than skipping correctness checks.

Common Mistakes

Assuming unchanged model bytes imply unchanged predictions.
Canarying only the dominant host class.
Calling a patched version safe without verifying the fixed build.
Planning rollback to the known vulnerable image as the only escape.