Ensemble release review project

Freeze the decision and data boundary

The service predicts clearance hours at intake so dispatch can allocate staff. Write the action threshold, cost of late underestimation, target clock and feature availability before training. Use shipment groups and a later date for final evaluation. Keep the later cohort untouched during feature design, tree sizing and stage selection. The validation lesson defines that separation.

Build one comparable scorecard

Fit the training-median baseline, a constrained tree, a bagged forest candidate and a boosted candidate on the same permitted training rows. Select settings using development validation. On the final cohort, store row-aligned predictions and report MAE, costly-underestimate rate, support by site and serving latency. A favorable out-of-bag score is useful diagnostic evidence but cannot replace the later cohort. Compare OOB behavior and selected boosting stage with their own limitations stated.

Quantify the uncertain gain

Calculate each candidate’s paired gain over the baseline and resample the independent evaluation groups. Report the observed gain, an interval, group count and tail cases. If the interval includes a material loss or one site drives the result, do not hide that inside an overall score. The paired interval guide supplies the mechanics; the release decision remains a policy choice.

Check the serving contract

Save model and preprocessing versions, feature order, missing-value policy, seed, training cutoff and selected settings. Reconstruct one historical request through the serving path and compare with the offline prediction. Set a latency budget and measure it under realistic batch size. Confirm that delayed outcomes are not treated as instant feedback. The monitoring guide separates immediate input alerts from mature error.

Record a disposition and next experiment

The code is a small decision gate over a reviewed scorecard; it cannot decide statistical validity by itself. State whether the candidate is ready for a limited pilot, needs more independent labels or fails a service constraint. Assign an owner and date to each blocker. If approved, plan a monitored comparison with rollback, rather than silently replacing the incumbent. The same record should explain why a simpler model was or was not sufficient.

Implementation

def release_disposition(review):
    blockers = []
    if not review["feature_clock_verified"] or not review["final_test_sealed"]:
        blockers.append("evaluation boundary")
    if review["independent_sites"] < 5:
        blockers.append("site support")
    if review["gain_interval_low_hours"] <= 0:
        blockers.append("uncertain gain")
    if review["tail_miss_rate"] > review["tail_rate_limit"]:
        blockers.append("costly tail")
    if review["p95_latency_ms"] > review["latency_budget_ms"]:
        blockers.append("serving latency")
    if not review["mature_label_monitor_ready"]:
        blockers.append("outcome monitoring")
    return "pilot review" if not blockers else "hold: " + ", ".join(blockers)

shipment_review = {
    "feature_clock_verified": True,
    "final_test_sealed": True,
    "independent_sites": 6,
    "gain_interval_low_hours": -0.2,
    "tail_miss_rate": 0.08,
    "tail_rate_limit": 0.06,
    "p95_latency_ms": 31,
    "latency_budget_ms": 45,
    "mature_label_monitor_ready": True,
}
decision = release_disposition(shipment_review)
assert decision == "hold: uncertain gain, costly tail"

Performance and operating cost

Scoring N final shipments with T tree nodes traversed per prediction has model-dependent cost; paired score and slice aggregation is O(N), while grouped bootstrap adds O(BN) in a direct implementation. Serving p95 latency, retraining time and monitoring storage must be measured in the target environment.

Common Mistakes

Do not promote a candidate using its out-of-bag score alone.
Do not count correlated scans as independent sites.
Do not announce a release when outcome monitoring is not ready.