Outcome misclassification: correct an observed rate only under stated label-error assumptions

Last updated: 7 Oct 20265 min read

tutorial

A verified sensitivity and specificity pair can adjust a binary observed rate when the labeling process is stable and sufficiently informative.

Separate event from recorded label

An invoice audit uses an automated flag as a proxy for confirmed duplicate payment. Some true duplicates are missed, and some valid invoices are flagged. The observed flag rate is not the true duplicate rate. Sensitivity is the chance of a flag given a true duplicate; specificity is the chance of no flag given a truly valid invoice. Both need an adjudicated validation set reflecting the same cases and operating conditions as the target period. The diagnostic lesson explains how prevalence changes what a positive flag means.

Understand the algebra and its boundary

Under constant sensitivity and specificity, observed positive rate equals sensitivity times true prevalence plus one minus specificity times one minus prevalence. Rearranging gives corrected prevalence as observed rate plus specificity minus one, divided by sensitivity plus specificity minus one. The denominator must be positive for this orientation to contain useful information. The code raises an error when the supplied values imply an impossible corrected fraction; silently clipping it to zero would hide an inconsistent model or noisy estimates.

Carry validation uncertainty

Sensitivity and specificity are estimates, not fixed facts. A rare event can make the corrected rate very sensitive to a small specificity error because false positives may outnumber true positives. Report the adjudication sample sizes and uncertainty in all inputs, then propagate it with an appropriate interval or resampling method. If the flagging process changed by branch, use branch-specific validation or a model that states its transport assumptions. The rare-rate lesson warns against overconfident intervals near zero.

Ask who was verified

If only flagged invoices are manually reviewed, specificity among unflagged invoices cannot be estimated. The algebra cannot manufacture that missing information. Sample some unflagged invoices for adjudication and record their selection probabilities. The verification lesson corrects the validation design; the project refuses a corrected headline until that sampling is documented.

Implementation

def corrected_binary_prevalence(observed_positive_rate,
                                sensitivity, specificity):
    if any(not 0 <= value <= 1 for value in
           (observed_positive_rate, sensitivity, specificity)):
        raise ValueError("inputs must be proportions")
    information = sensitivity + specificity - 1
    if information <= 0:
        raise ValueError("label test is not informative in this orientation")
    corrected = (observed_positive_rate + specificity - 1) / information
    if not 0 <= corrected <= 1:
        raise ValueError("supplied rates are incompatible with this model")
    return corrected

assert round(corrected_binary_prevalence(0.08, 0.80, 0.96), 6) ==        round(0.04 / 0.76, 6)

Performance and operating cost

The algebra is O(1). Estimating sensitivity and specificity requires adjudication, and uncertainty propagation costs additional computation. When specificity is near one and event prevalence is tiny, a minor validation error can dominate the corrected answer.

Common Mistakes

Calling a flag a confirmed outcome.
Using sensitivity and specificity from another policy version without a transport check.
Clipping an impossible corrected rate instead of investigating model inconsistency.
Treating estimated error rates as exact when calculating final uncertainty.