Project: detect a stale-feature failure in a cold-chain model

Last updated: 6 Oct 20265 min read

project

Build isolated production probes, map route coverage and rehearse a failure that a simple health check misses.

Register the probe suite

The cold-chain service predicts whether refrigerated shipments require quarantine. Register an ordinary reading, an expired feature record and a boundary reading near the policy cutoff. Each fixture carries a revision, expected route envelope, model and feature versions, region and deadline. Mark requests as synthetic through a trusted internal identity; customer clients cannot add the marker. The probe contract prevents synthetic outcomes from entering the claims and training ledgers.

Expose the misleading green light

The HTTP readiness endpoint still returns success while the online feature store serves a snapshot from the prior day. The ordinary fixture uses a cached default and passes, but the stale-feature fixture should invoke quarantine fallback. Its route trace reveals that the first probe never fetched the feature store. Update the route coverage ledger; a green status without that dependency was incomplete. Probe coverage is defined by executed paths, not fixture count.

Exercise alert and rollback

Point a dedicated canary at the stale snapshot, confirm two consecutive failures and page the serving owner. An unsafe release result pages immediately. Restore the last approved feature pointer, rerun the probe and inspect production feature freshness before closing the event. Keep the probe output out of real shipment release and outcome labels throughout the drill. Record exact timestamps, revisions and decisions in the incident timeline.

Handoff an honest report

Report which routes the suite covers, which it does not, the delayed first detection, false alerts, regional run counts and fixture expiry dates. A passing probe does not verify statistical model quality, so schedule a separate mature-outcome review. Drift monitoring addresses that slower question. The completed project has a runnable route gate, an alert record and a tested rollback owner.

Implementation

def cold_chain_probe_gate(trace, expected):
    missing = expected["required_stages"] - set(trace["stages"])
    if missing:
        return "fail:missing-stage"
    if trace["feature_age_min"] > expected["max_feature_age_min"]:
        return "fail:stale-feature"
    if trace["route"] != expected["route"]:
        return "fail:route"
    return "pass"

expected = {"required_stages": {"parse", "feature-read", "model", "policy"},
            "max_feature_age_min": 47, "route": "quarantine"}
trace = {"stages": ["parse", "feature-read", "model", "policy"],
         "feature_age_min": 82, "route": "quarantine"}
assert cold_chain_probe_gate(trace, expected) == "fail:stale-feature"
assert cold_chain_probe_gate({**trace, "feature_age_min": 39}, expected)        == "pass"

Performance and operating cost

Stage comparison is O(s) time and O(s) extra space for s stages. Probe cost is a small stream of inference calls and traces; the rehearsal adds controlled failure work. A stale feature that goes undetected can invalidate many customer decisions, so the probe budget should match the cost and time sensitivity of that failure.

Common Mistakes

Testing only process readiness while skipping the feature read.
Letting probe requests trigger real shipment release.
Calling the incident closed after the probe passes without checking customer decisions.
Leaving fixtures tied to an obsolete model revision.