Project: audit support resolution with open cases and imperfect labels

Last updated: 5 Oct 20265 min read

project

Produce a reviewable cohort analysis that preserves unresolved cases, audits a closure proxy and states when the decision is sensitive.

Freeze the intake cohort

Create 162 eligible support cases across email, chat and self-service channels. Give every case a stable ID, submission time, channel, priority and analysis cutoff. Include 11 out-of-scope spam records in the raw fixture but not the eligible denominator, plus duplicate status events that must not create duplicate cases. Version the eligibility and exclusion rules. Cohort entry must be reproducible from the raw event log.

Track the observation clock

For each eligible case, store observed duration and whether confirmed resolution occurred by cutoff. Include 19 cases without final confirmation, some open and some with missing customer response. Build an unresolved-share curve and an at-risk table through the two-day mark; verify tie handling against a hand-calculated fixture. Do not replace missing final times with the cutoff duration in an ordinary average. The censoring lesson gives the calculation contract.

Audit the outcome proxy

Select a fixed review sample across channels. Keep ticket-closed and reviewer-confirmed-solved as separate fields, including unknown review outcomes. Produce paired-label counts and disagreement by channel. Record reviewer guidance and adjudication notes. If chat cases use an automatic close rule, show its effect on the proxy separately from the underlying confirmed outcome. The proxy audit determines what the closure field can support.

Write the decision packet

Report the intake count, exclusions, follow-up-eligible count, observed two-day successes, unresolved cases, proxy disagreements and any missing frame segment. Calculate a lower and upper rate under missing outcomes, then test a 60% operating gate. State whether the gate can flip; if it can, specify the follow-up audit that would be worth doing next. Do not replace the range with one favorable point estimate.

Verify the work

Save the raw fixture, code revision, snapshot cutoff, cohort ledger, review sample selection, scenario parameters and outputs. Include assertions that eligible equals resolved plus open at cutoff; that paired review counts equal all selected review records; and that each scenario preserves the original cohort denominator. A peer should reproduce every headline count without a dashboard or private database.

Implementation

def decision_state(success_count, observed_count, unresolved_count, required_rate):
    if not 0 <= success_count <= observed_count or unresolved_count < 0:
        raise ValueError("invalid counts")
    total = observed_count + unresolved_count
    if total == 0:
        raise ValueError("empty cohort")
    lower = success_count / total
    upper = (success_count + unresolved_count) / total
    return {"lower_rate": lower, "upper_rate": upper,
            "decision_stable": lower >= required_rate or upper < required_rate}

assert decision_state(86, 143, 19, 0.60)["decision_stable"] is False

Performance and operating cost

Cohort construction and paired-label reconciliation scan N events in O(N) expected time with ID indexes and O(N) storage. Sorting event durations for the unresolved-share curve costs O(N log N). Reviewer labor and delayed follow-up dominate compute cost; budget them explicitly.

Common Mistakes

Do not silently discard cases without final confirmation.
Do not collapse closure and confirmed resolution into one field.
Do not publish a pass or fail when plausible missing outcomes cross the operating gate.