Project: audit a refund service result with missing final outcomes

Last updated: 7 Oct 20265 min read

project

Rebuild eligible refund cohorts, audit observation probabilities, and show what unobserved outcomes could change.

Freeze the cohort

A service report claims the on-time refund rate rose after a queue change. Define approved refunds entering before the reporting cutoff, the service-time clock, outcome maturity, and every final-status source. Keep records that lack a final outcome in the eligible frame. The first dashboard selected only completed refunds, a condition affected by the process under review. The missing-outcome ledger separates eligibility, maturity, and observation.

Trace an uneven feed

A branch migrated its status feed midway through the study. High-priority refunds are more likely to be missing after the migration, so complete cases look faster even if service did not improve. Reconcile IDs across systems and compare observation rates by branch, week, priority, and queue. A segment with no observed final outcomes cannot be recovered by inverse weighting alone. Hold the estimate until support and source errors are addressed. The weighting lesson makes this limit visible.

Calculate several honest views

Produce the observed-case rate, a weighted rate under a stated measured-data observation model, and a range under worse outcomes among missing cases. Include uncertainty from branch grouping and fitted observation probabilities. The figures answer different questions and should not be averaged into a single release number. If the weighted estimate and plausible missing-outcome scenarios disagree about the target, the dashboard cannot claim a settled improvement. Bounds show the broadest assumption-light range.

Deliver a gated decision

The packet includes the eligible frame, outcome maturity, source reconciliation, missingness reasons, observation model, probability support, sensitivity assumptions, and branch-aware uncertainty. The gate below blocks a public claim if any structural absence or unmatched feed remains. It does not certify the weighted estimate; statistical review must still judge the observation assumption. The clustering lesson identifies the independent branch unit for uncertainty.

Implementation

def refund_missingness_gate(audit, limits):
    if audit["unmatched_status_ids"]:
        return "hold:source-reconciliation"
    if audit["unobserved_priority_strata"]:
        return "hold:positivity"
    if audit["minimum_observation_probability"] < limits["minimum_probability"]:
        return "review:unstable-weights"
    if not audit["missing_outcome_sensitivity_done"]:
        return "hold:sensitivity"
    return "review:scoped-service-result"

limits = {"minimum_probability": 0.12}
audit = {"unmatched_status_ids": 2, "unobserved_priority_strata": 0,
         "minimum_observation_probability": 0.28,
         "missing_outcome_sensitivity_done": True}
assert refund_missingness_gate(audit, limits) == "hold:source-reconciliation"
assert refund_missingness_gate({**audit, "unmatched_status_ids": 0},
                               limits) == "review:scoped-service-result"

Performance and operating cost

The gate is O(1). Reconstructing statuses is O(n) expected time with keyed lookups, while model fitting and resampling add cost. A dashboard built only from completed cases is faster because it discards the very outcomes most capable of changing the claim.

Common Mistakes

Treating unresolved statuses as successful refunds.
Fitting weights for a stratum with no observed outcomes.
Publishing one weighted rate without its missing-data assumption.
Ignoring branch-level correlation when reporting uncertainty.