Distribution shift response project

Last updated: 5 Oct 20265 min read

project

A release review joins input support, mature outcome change, time-correct retraining, paired shadow evidence and rollback into one decision packet.

Set the decision boundary

A new depot changes the mix of shipment backlogs. Define the intake feature snapshot, handoff target, label-ready rule, staffing deadline and affected sites before examining candidate performance. Preserve case IDs across training, shadow and final evaluation. The support audit determines whether the old model even covers the new traffic.

Separate evidence types

Input-shift counts arrive promptly; error and calibration require mature outcomes. Importance weighting can estimate target-mix risk only under overlap and a stable within-input outcome mechanism. A within-slice error change challenges that premise. Report these as separate findings rather than one drift score. Weighted risk and concept checks make the boundary explicit.

Replay the proposed change

Use rolling origins with correct label availability to compare fixed and retrained policies. Hold out a later period that did not select the schedule, features, weights or thresholds. Run a challenger in shadow on the same new intakes and log missing scores. The backtest and shadow comparison answer different questions.

Audit the operating consequences

Check error by site, calibration, false negatives, reviewer capacity, latency and the cost of a false alert. Include unsupported new cases in the report and route them through a declared fallback. A positive average paired loss difference does not establish a good outcome for every group or action. Group errors and review load are release inputs.

Make the disposition explicit

The code checks whether required artifacts are present; it does not judge whether their results meet a chosen business threshold. Attach measured values, decision owners, an approved pilot window, the old model version and a rollback trigger. If labels have not matured, keep the decision open instead of treating missing evidence as a pass.

Implementation

def shift_review_gate(packet):
    requirements = {
        "support_checked": "target support",
        "label_clock_documented": "label clock",
        "future_holdout_sealed": "future holdout",
        "shadow_missingness_reported": "shadow missingness",
        "group_errors_reported": "group errors",
        "review_capacity_checked": "review capacity",
        "rollback_version_retained": "rollback version",
    }
    missing = [label for field, label in requirements.items() if not packet.get(field)]
    return "ready for pilot review" if not missing else "hold: " + ", ".join(missing)

depot_packet = {
    "support_checked": True, "label_clock_documented": True,
    "future_holdout_sealed": True, "shadow_missingness_reported": False,
    "group_errors_reported": True, "review_capacity_checked": False,
    "rollback_version_retained": True,
}
assert shift_review_gate(depot_packet) == (
    "hold: shadow missingness, review capacity"
)

Performance and operating cost

The checklist costs O(K) for K requirements. Cohort counting and paired scoring are O(N); R rolling model fits may cost roughly R times a single fit, depending on the estimator. The substantial operating costs are archived availability data, mature outcomes, shadow serving and a credible rollback path.

Common Mistakes

Do not call an input-distribution alarm a measured accuracy loss.
Do not promote a challenger on a shadow score alone when actions change outcomes.
Do not pass an incomplete decision packet by treating absent labels or capacity as zero risk.