Multi-stage inference: pin each stage and its contract

Last updated: 6 Oct 20265 min read

tutorial

A serving chain needs versioned stage inputs and outputs so a healthy final endpoint cannot hide a broken intermediate transform.

Describe the whole chain

A receipt request may pass through image extraction, merchant normalization, risk scoring and a route policy. The final risk model alone does not define the behavior; a changed extraction model can alter every downstream feature. Record the stage order, artifact digests, input and output schemas, policy revision and overall chain revision. Training-serving parity compares feature meaning, while the public API contract covers only what callers send and receive. Both matter, but neither fully describes intermediate stage identity.

Specify stage failure states

Each stage should return a typed success or a named failure such as unreadable image, unsupported currency, timeout or contract mismatch. Do not turn missing text into an empty string that happens to produce a plausible low-risk score. Decide which failures route to manual review and which reject the request. A stage may be optional only if the downstream model was evaluated with that stage absent. The approved scope should name that degraded path; a fallback is a different decision policy, not invisible error handling.

Allocate one end-to-end deadline

Give each stage a sub-budget within the caller deadline and retain queue, network and serialization time. A chain of individually fast stages can still miss the endpoint SLO, especially when a shared feature service stalls. Pass the remaining deadline downstream rather than starting a fresh full timeout at every hop. Tail latency measurement needs stage-level spans and one end-to-end clock. Count a stage timeout as a failed or degraded decision even if a later policy returns a successful manual-review route.

Promote as a tested unit

An independent extractor release can change the risk model input distribution while its digest stays fixed. Test stage contracts, a frozen end-to-end fixture set and a small live canary before changing the chain pointer. Preserve the previous complete chain for rollback; mixing old and new stages during rollback can produce an untested combination. The project injects an extractor change and verifies that the route policy receives a typed failure rather than a fabricated low-risk score.

Implementation

def chain_decision(extraction, risk_score, chain):
    if extraction["digest"] != chain["extractor_digest"]:
        return {"route": "hold", "reason": "extractor-mismatch"}
    if extraction["state"] != "ready":
        return {"route": "manual-review", "reason": extraction["state"]}
    if risk_score is None or not 0 <= risk_score <= 1:
        return {"route": "hold", "reason": "invalid-risk-score"}
    return {"route": "review" if risk_score >= chain["review_at"]
            else "clear", "reason": "scored"}

chain = {"extractor_digest": "extract-47", "review_at": 0.72}
extraction = {"digest": "extract-47", "state": "ready"}
assert chain_decision(extraction, 0.81, chain)["route"] == "review"
assert chain_decision({**extraction, "state": "unreadable"}, None, chain)[
    "route"] == "manual-review"

Performance and operating cost

The illustrated gate is O(1) time and space. A chain with s serial stages has latency near the sum of stage service and transfer times, with each stage able to fail the request. Extra stage replicas, contract tests and complete-chain rollback packages raise operating cost but prevent an untested combination of otherwise valid artifacts.

Common Mistakes

Logging only the final model digest.
Mapping extraction failure to an ordinary empty feature.
Giving every stage a full endpoint timeout.
Rolling back one stage into a chain combination never tested.