Correct a binary outcome rate for label error

Last updated: 5 Oct 20265 min read

tutorial

When sensitivity and specificity are known for the target population, an observed binary-label rate can be inverted to estimate the underlying outcome rate if their sum exceeds one.

Write the observation process first

Let q be the fraction of tickets production flags as breaches and p the adjudicated breach fraction in the same population. If production sensitivity is Se and specificity is Sp, then q equals Se times p plus (1 minus Sp) times (1 minus p). Solving gives p = (q + Sp - 1) / (Se + Sp - 1). The identity depends on the error rates applying to this exact ticket population and period. Validation estimates supply Se and Sp.

Check whether inversion is possible

When Se plus Sp is one, the production flag carries no prevalence information under this model. When their sum is below one, the classifier is worse than its flipped version; the simple positive-orientation correction is not appropriate without reconsidering the label definition. Even with a positive denominator, a proposed pair of error rates can imply p below zero or above one for the observed q. That is an incompatible assumption set, not a value to silently clip.

Use the same population for every quantity

A production rate for September cannot safely be corrected with error rates measured only in February after a logging rewrite. Match ticket eligibility, branch, time and reference-label rubric. A validation sample selected on production label needs design-aware error rates. The correction yields an estimated descriptive rate, not a causal policy effect and not a confidence interval. Group contrasts need separate care.

Understand why the change can be large

When breaches are rare, a small false-positive rate can form a substantial share of observed positives. In the fixture, q is 0.18, Se is 0.90 and Sp is 0.95; the corrected p is about 0.153. A one-point change in assumed specificity may move p more than a one-point change in sensitivity. Inspect a plausible range rather than fixating on one estimate. The assumption grid does that.

Carry uncertainty separately

The equation treats Se and Sp as fixed inputs, but validation estimates have sampling uncertainty, and q does too. A formal interval would propagate all three quantities and any dependence from an internal validation sample. The code intentionally only checks the algebra and feasibility. A corrected point rate with many decimal places can mislead when the audit sample contains few reference positives.

Implementation

def corrected_binary_rate(observed_rate, sensitivity, specificity):
    if not all(0 <= value <= 1 for value in
               (observed_rate, sensitivity, specificity)):
        raise ValueError("rates must lie between zero and one")
    information = sensitivity + specificity - 1
    if information <= 0:
        raise ValueError("uninformative or reversed label process")
    corrected = (observed_rate + specificity - 1) / information
    if not 0 <= corrected <= 1:
        raise ValueError("observed rate incompatible with assumptions")
    return corrected

production_breach_rate = .18
estimated_true_rate = corrected_binary_rate(production_breach_rate, .90, .95)
assert abs(estimated_true_rate - (.13 / .85)) < 1e-12
assert estimated_true_rate < production_breach_rate

Performance and operating cost

One inversion is O(1) time and space. Estimating group-specific accuracy and propagating validation uncertainty dominate the workflow. The correction is numerically unstable when sensitivity plus specificity approaches one.

Common Mistakes

Do not clip an infeasible corrected rate into the valid range.
Do not reuse error rates from a different population without a transport argument.
Do not call a point correction an uncertainty interval or a causal effect.