Project: separate claims-flagger errors from reviewer disagreement

Construct an observable truth frame

A duplicate-claim flagger is evaluated on cases auditors already reviewed. That selected set contains many flagged claims and almost no unflagged cases, so it cannot estimate false negatives in the full submission stream. Predeclare a random audit sample from both flag statuses, retain selection probabilities and mature outcomes, and isolate reviewers from the model score where feasible. Diagnostic quantities require all four confusion cells.

Audit the labels themselves

Two reviewers independently classify the sampled claims. Their overall agreement is high because most cases are legitimate, but the positive-label cells have more disputes. Preserve both original labels and send disagreement cases through documented adjudication. A kappa value alone does not say which reviewer is correct. The agreement lesson ties each statistic to the full cross-tabulation and prevalence.

Estimate operational yield

Use audited confusion counts with the sampling design to estimate sensitivity, specificity and live positive predictive value. At rare duplicate prevalence, even a high-sensitivity flagger can produce many false alerts. Show alert volume and missed-duplicate counts under the proposed threshold; report uncertainty on sparse positive cells. Do not use the oversampled audit fraction as the live prevalence. Weighting logic links sampled cases to the eligible claim frame.

Publish a gated packet

Require adequate unflagged audit coverage, a resolved label policy, mature outcomes and enough positive cases before a broad performance claim. Record the reviewer disagreement matrix, adjudication rule, confusion counts, selection probabilities, estimated workload and error-cost owner. If any component fails, describe the audit as incomplete. Observation bias and rare-event intervals define the next collection step.

Implementation

def claims_audit_gate(report, limits):
    if report["unflagged_audited"] < limits["minimum_unflagged"]:
        return "hold:unflagged-coverage"
    if report["unresolved_disagreements"]:
        return "hold:label-adjudication"
    if report["mature_positive_cases"] < limits["minimum_positives"]:
        return "hold:rare-positive-support"
    if not report["selection_probabilities_recorded"]:
        return "hold:sample-design"
    return "publish:scoped-diagnostic-review"

limits = {"minimum_unflagged": 52, "minimum_positives": 17}
report = {"unflagged_audited": 34, "unresolved_disagreements": 0,
          "mature_positive_cases": 21,
          "selection_probabilities_recorded": True}
assert claims_audit_gate(report, limits) == "hold:unflagged-coverage"
assert claims_audit_gate({**report, "unflagged_audited": 63},
                         limits) == "publish:scoped-diagnostic-review"

Performance and operating cost

The gate is O(1) time and space once counts are available. Randomly auditing unflagged cases and resolving reviewer disagreements consume staff time but are needed to estimate false negatives and live alert yield. Reusing only previously reviewed claims would be cheaper and would leave the main error rate unidentified.

Common Mistakes

• Treating unreviewed claims as true negatives.
• Using an oversampled audit prevalence as the live base rate.
• Discarding reviewer disagreements from the confusion matrix.
• Publishing one accuracy figure without denominators and workload estimates.

Read next

• Diagnostic performance: separate sensitivity, specificity and predictive value
• Reviewer agreement: inspect confusion cells before one kappa number
• Rare proportions: keep interval uncertainty visible at zero and one
• Stratified survey estimates: weight toward the named population
• Selective labels: measure what the model never lets reviewers see

Continue the workflow: Project: compare claims centers under a shared severity mix.

Continue the workflow: Project: estimate duplicate-invoice prevalence with audited label error.