Model cascades: pin component versions and route decisions

Last updated: 6 Oct 20265 min read

tutorial

A fast first-stage model and a slower specialist must be released as one routed decision system.

Name the cascade as the release unit

A receipt pipeline may run a cheap quality detector first, then send uncertain cases to a richer risk model. The final decision depends on both model digests, preprocessing, the uncertainty cutoff and the route policy. Promoting either model alone changes the effective system. Create a cascade manifest containing every component, input and output schema, routing rule, timeout and approved fallback. Inference chain identity records the execution path; the cascade manifest records the permitted combinations.

Preserve one decision context

Every stage should receive the same immutable receipt snapshot or an explicitly versioned derived view. Carry a decision ID, component versions and feature snapshot across the handoff. If the first stage times out, do not treat its absent score as “low risk”; choose a documented fallback or manual route. If the second stage fails, preserve the first-stage result and error state without inventing a final score. Deadline tests expose these cases.

Measure routed populations

The specialist only sees cases selected by the first stage, so its observed quality and latency cannot be compared to the full population without route coverage. Evaluate the complete cascade on a frozen end-to-end set, then inspect each branch, threshold boundary and cohort. Count manual routes and missing outcomes. Selective labels matter when a branch changes which receipts receive review.

Roll out a compatible pair

Preload all required component artifacts and prove schema and runtime compatibility before moving the cascade pointer. Stage a sticky cohort, observe branch proportions and tail latency, then expand. Keep the prior complete manifest as rollback; rolling back only one component can create an untested pair. The budget gate protects tail cost, and the project catches a schema mismatch hidden by a happy-path smoke test.

Implementation

def cascade_route(quality_score, quality_model, risk_model,
                  review_threshold=0.64):
    if not quality_model or not risk_model:
        raise ValueError("component digests required")
    if quality_score is None:
        return {"route": "manual-review", "reason": "quality-unavailable"}
    if not 0 <= quality_score <= 1:
        raise ValueError("invalid score")
    return {"route": "risk-model" if quality_score < review_threshold
            else "fast-accept", "quality_digest": quality_model,
            "risk_digest": risk_model}

assert cascade_route(0.47, "quality-r8", "risk-r31")["route"] == "risk-model"
assert cascade_route(None, "quality-r8", "risk-r31")["route"] == "manual-review"

Performance and operating cost

Routing is O(1) time and space per request. Total compute is first-stage cost plus specialist cost multiplied by route share; tail latency also includes handoff and fallback. A lower specialist share can save compute yet miss difficult receipts, so route changes need end-to-end quality gates rather than a cost-only target.

Common Mistakes

Promoting one component while leaving an incompatible partner live.
Interpreting a missing first-stage score as a safe result.
Reporting specialist accuracy as if it covered every receipt.
Rolling back one digest instead of the complete cascade manifest.