Project: decide whether a service recovery target survives missing outcomes

Last updated: 7 Oct 20265 min read

project

Reconcile repair cohorts, pool imputed analyses, and test how much slower unseen closures could be before a release claim fails.

Freeze the service question

A repair team claims a queue change reduced resolution time for urgent cases. Define eligible tickets, the clock start and stop, which tickets have matured, the comparison period, and the predeclared improvement threshold. Keep a ticket without a final timestamp in the eligible denominator. If the new queue also changes which cases receive a final scan, analyzing completed tickets alone conditions on the intervention. The cohort ledger must reconcile before modeling.

Build and inspect several completed datasets

Use baseline branch, device type, priority, intake period, and other pre-resolution variables to draw plausible missing times. Restrict impossible durations. Review observed-versus-filled distributions by urgent status and branch; mask a prespecified set of known cases for a reconstruction check. Fit the same urgent-case contrast in each completed dataset and combine coefficient and variance outputs. A model that imputes urgent tickets as routine tickets simply because the routine group is larger is not acceptable. Combination rules carry both variance sources.

Challenge the favorable model

Add extra delay to imputed urgent cases over a stated range and repeat the analysis. Show the tipping shift that would erase the target improvement. If a branch has no observed urgent outcomes, there is no direct evidence for its timing distribution; a pooled model may extrapolate, but that should be labeled and reviewed. The sensitivity lesson explains why a good masked-observation score cannot settle this risk.

Release a decision packet

Include cohort counts, missing reasons, branch-level observation rates, variables and constraints used in imputation, draw count, analysis model, pooled estimate, uncertainty, and the shift grid. The gate below blocks an unreviewed decision when missing urgent outcomes, diagnostics, or sensitivity work are absent. A passing gate is a documentation condition; it is not statistical proof that the unseen cases behave like observed ones. Preserve versioned extracts so the result can be reproduced.

Implementation

def recovery_release_gate(audit):
    if audit["eligible"] != audit["observed"] + audit["missing"]:
        return "hold:cohort-reconciliation"
    if audit["missing_urgent"] and not audit["urgent_diagnostics_passed"]:
        return "hold:urgent-imputation"
    if audit["missing"] and not audit["sensitivity_grid_reviewed"]:
        return "hold:unseen-outcome-shift"
    return "review:effect-and-uncertainty"

audit = {"eligible": 47, "observed": 41, "missing": 6,
         "missing_urgent": 3, "urgent_diagnostics_passed": False,
         "sensitivity_grid_reviewed": True}
assert recovery_release_gate(audit) == "hold:urgent-imputation"
assert recovery_release_gate({**audit, "urgent_diagnostics_passed": True})        == "review:effect-and-uncertainty"

Performance and operating cost

The gate is O(1); the material cost is rebuilding ticket lineage and rerunning the model for each completed dataset and sensitivity scenario. Cheap complete-case reporting cannot recover tickets missing precisely because resolution was difficult.

Common Mistakes

Calling filled timestamps observed facts.
Imputing with fields recorded only after closure.
Suppressing branches with no observed urgent outcomes.
Publishing only the favorable missing-at-random result without its tipping shift.