Project: audit a weighted contact-center satisfaction estimate

Last updated: 7 Oct 20265 min read

project

Rebuild the population-weighted rate, expose a single-branch stratum and hold an overconfident statewide interval.

State who is represented

A contact center wants a monthly satisfaction number across metropolitan and rural cases. Record all eligible closures, invitation mode, question wording, region population totals and the response window. The rural region was deliberately oversampled, so the raw respondent average is not the combined population rate. The stratified estimate uses eligible-case shares while preserving each region’s observed rate.

Audit the response trail

Count invited, contacted, completed and item-missing cases by region and branch. In the first export, rural cases came from one branch; a contractor also failed to transmit the question for a set of night contacts. Keep those misses visible. A region weight does not restore answers that are systematically missing inside that region. Compare response patterns and state a bounded sensitivity scenario before any statewide claim. The observation process is part of the result.

Reject false precision

Compute the weighted point estimate from the documented population totals, then inspect distinct sampled branches per stratum and weight concentration. The one-branch rural stratum cannot support the advertised ordinary row-level standard error. Hold that interval, consult the sampling plan for an approved variance method and gather more branches if the desired precision requires them. The design check distinguishes respondent count from independent branch support.

Deliver a reviewable packet

Report regional rates with completed denominators, population controls and dates, sample-selection steps, contact and item nonresponse, branch counts, weight summary and interval method. Mark any unmeasured region or stratum explicitly. The point estimate may be presented with a clear scope if the data owner approves, but do not attach an unsupported confidence claim. Coverage language and the weighting audit connect to the follow-up collection plan.

Implementation

def survey_publication_gate(report, limits):
    if report["unmatched_population_regions"]:
        return "hold:population-frame"
    if report["missing_item_share"] > limits["maximum_missing_share"]:
        return "review:item-nonresponse"
    if min(report["branches_per_stratum"].values()) < 2:
        return "hold:variance-support"
    if report["weight_effective_count"] < limits["minimum_effective_count"]:
        return "hold:weight-concentration"
    return "publish:design-reviewed"

limits = {"maximum_missing_share": 0.08, "minimum_effective_count": 72}
report = {"unmatched_population_regions": False, "missing_item_share": 0.04,
          "branches_per_stratum": {"metro": 4, "rural": 1},
          "weight_effective_count": 86}
assert survey_publication_gate(report, limits) == "hold:variance-support"
assert survey_publication_gate({**report, "branches_per_stratum":
       {"metro": 4, "rural": 3}}, limits) == "publish:design-reviewed"

Performance and operating cost

The gate scans g strata in O(g) time and O(g) temporary space. A proper interval needs the sampling plan, primary-unit identifiers and design-aware software; extending collection across rural branches costs more than calculating a weighted point estimate. Publishing an ordinary row-level interval would be cheap but unjustifiably precise.

Common Mistakes

Calling a raw respondent average the statewide satisfaction rate after rural oversampling.
Treating a missing survey answer as a negative satisfaction response.
Computing an ordinary standard error despite a one-branch rural stratum.
Dropping branch and selection metadata from the publication packet.