Project: review resolution predictions with competing exits

Fix a first-event ledger

Set case creation as time zero and choose day nine as the business horizon. For each case, keep one first terminal event: resolved or cancelled. Record last observed age for still-open cases and a distinct loss status for records whose feed stopped early. Reconcile this case ledger to the raw event log and document timestamp corrections. The outcome contract prevents cancellations from disappearing into a generic censor field.

Estimate cohort incidence

Build resolution and cancellation cumulative-incidence curves from the same arrival cohort. Show event counts and risk-set counts beside the curves, including how many cases were lost before day nine. Check that resolution incidence, cancellation incidence and event-free share sum to one at each event time. Do not use one minus a resolution-only Kaplan–Meier curve as observed resolution probability when cancellations block resolution. The incidence lesson provides the arithmetic fixture.

Review the prediction vector

For each case, freeze nine-day resolution and cancellation probabilities at creation. Reject negative probabilities and any pair summing above one. Inspect predicted versus observed first-event shares, and score resolution with the same known-horizon cohort used for calibration. If early loss exists, use a documented censor-aware method and inspect support. Cause-specific rates can be combined into coherent incidence probabilities.

Compare a later-period holdout

Evaluate on later arrivals with a mature nine-day horizon. Keep both baseline and revised predictions for the same cases, then calculate their paired score difference and uncertainty at the correct resampling unit. If one organization contributes repeated cases, resample that organization rather than pretending cases are independent. Document cancellation-policy changes across periods. The paired bootstrap conditions on frozen predictions.

Withhold unsupported claims

The compact gate below rejects probability-budget failures and reports unknown horizon states; it does not fit incidence curves or approve a model. The release packet should state the target, cohort dates, first-event counts, early-loss count, curve support, calibration, score comparison, interval method and subgroup coverage. Publish no nine-day accuracy figure from the simple path while unknown states remain.

Implementation

def competing_exit_review_gate(case_forecasts, horizon_days):
    if horizon_days <= 0 or not case_forecasts:
        raise ValueError("invalid review cohort")
    if len({case_id for case_id, *_ in case_forecasts}) != len(case_forecasts):
        raise ValueError("duplicate case ID")
    counts = {"resolved": 0, "cancelled": 0,
              "open_at_horizon": 0, "unknown_at_horizon": 0}
    for case_id, resolution_risk, cancellation_risk, day, status in case_forecasts:
        if (not 0 <= resolution_risk <= 1 or
                not 0 <= cancellation_risk <= 1 or
                resolution_risk + cancellation_risk > 1 or day < 0):
            raise ValueError("invalid forecast or age")
        if status not in {"resolved", "cancelled", "open", "lost"}:
            raise ValueError("invalid case status")
        if status in {"open", "lost"} and day < horizon_days:
            counts["unknown_at_horizon"] += 1
        elif status == "resolved" and day <= horizon_days:
            counts["resolved"] += 1
        elif status == "cancelled" and day <= horizon_days:
            counts["cancelled"] += 1
        else:
            counts["open_at_horizon"] += 1
    counts["simple_score_ready"] = counts["unknown_at_horizon"] == 0
    return counts

forecast_ledger = [("E601", .63, .21, 4, "resolved"),
                   ("E602", .35, .42, 6, "cancelled"),
                   ("E603", .51, .16, 12, "open"),
                   ("E604", .41, .31, 5, "lost")]
review = competing_exit_review_gate(forecast_ledger, 9)
assert review["resolved"] == 1 and review["cancelled"] == 1
assert review["open_at_horizon"] == 1
assert review["unknown_at_horizon"] == 1
assert not review["simple_score_ready"]

Performance and operating cost

For N cases, the gate costs O(N) expected time and O(N) space for uniqueness checking. Incidence estimation, censor-aware evaluation and bootstrap intervals have additional costs. The gate is a data-contract check, not statistical validation.

Common Mistakes

Do not report a nine-day Brier score from a cohort containing unknown nine-day states without an explicit censor-aware method.
Do not compare models on different case sets.
Do not hide a cancellation-policy change inside an apparent model-quality trend.