Project: estimate support-case defects from an auditable sample

Last updated: 5 Oct 20265 min read

project

Build a sampling packet that a reviewer can use to reconstruct eligibility, selection, response and a weighted estimate.

Freeze the frame

Collect one row per eligible support case for a fixed quarter. Include intake ID, channel, submission timestamp, eligibility rule and a snapshot identifier. Compare the intake event log with the agent export and list uncovered self-service or failed-assignment cases. Deduplicate case IDs before sampling. The frame audit is a release gate, not an appendix to a rate.

Draw and record

Split the frame into main and partner channels. Choose random cases within each stratum and save the seed, chosen IDs, population counts and inclusion probabilities. Oversample the partner channel if it is small, but do not pool outcomes without design weights. Assign each selected case to a blind reviewer and preserve a distinct missing-outcome state for inaccessible evidence.

Analyze response and precision

Report response by channel, weighted defect rates, selected-sample bounds for unresolved labels and a design-aware uncertainty estimate. Compare the decision with and without a random follow-up of nonrespondents. Recompute the result from a frozen input manifest. Response assumptions and precision limits must appear beside the final number.

Submit failure evidence

Add a duplicate intake event, a case omitted from the export and one reviewed case with an attachment that cannot be opened. The pipeline should detect each without silently classifying it as non-defective. Deliver the frame diff, sample manifest, reviewer ledger, weighted calculation, interval procedure and a short decision memo that states exactly which population the estimate describes.

Implementation

def weighted_defect_fraction(review_rows):
    reviewed = [row for row in review_rows if row["defective"] is not None]
    total_weight = sum(row["design_weight"] for row in reviewed)
    if total_weight <= 0:
        raise ValueError("no observed weighted outcomes")
    return sum(row["design_weight"] * int(row["defective"])
               for row in reviewed) / total_weight

Performance and operating cost

The reviewed-row estimate is O(N) time and O(N) space as written; streaming accumulators reduce space to O(1). It describes reviewed units under the supplied weights, while nonresponse and coverage require separate adjustments or limits.

Common Mistakes

Do not suppress uncovered frame units in the final report.
Do not count missing reviews as clean cases.
Do not present a weighted respondent rate as assumption-free population truth.