Label drift: delayed outcomes and sampled quality audits

Last updated: 5 Oct 20265 min read

tutorial

A label pipeline can degrade when the source mix, guideline, interface or outcome delay changes.

Separate three changes

A new receipt layout changes evidence; an updated rejection policy changes the meaning of a label; and a slower refund process delays ground truth. Monitor these separately. Compare reviewer disagreement, unknown rates and adjudication reasons by source and week. Delayed-label monitoring explains why a current cohort may look clean simply because its failures have not arrived.

Audit a stable frame

Sample from all eligible receipts before escalation or model routing. Retain source, date, guideline and selection probability so a weighted estimate can be constructed for the intended population. Oversample rare scanner types for diagnosis, but weight or report them separately when estimating overall error. Do not mix an investigation sample with an independent audit in one headline metric.

Account for reviewer fatigue

Measure time per decision, repeated-item consistency and disagreement by reviewer shift. A spike in unknown labels may reflect worse images or exhausted reviewers. Rotate audit items and periodically recheck blinded stable fixtures, but avoid making them so recognizable that reviewers memorize answers. Escalate recurring rule gaps to the guideline owner with examples and a proposed revision.

Run an alarm review

Suppose unknown labels climb from 4 of 47 to 13 of 47 after a new scanner deployment. Check evidence availability, assignment mix and guideline version before declaring model drift. Pull a source-stratified blind sample, adjudicate disagreements and report whether the new scanner changed the item population, image quality or reviewer interpretation.

Implementation

def unknown_rate(review_rows, source):
    selected = [review for review in review_rows if review["source"] == source]
    if not selected:
        return None
    return sum(review["label"] == "unknown" for review in selected) / len(selected)

Performance and operating cost

A source-filtered pass costs O(N) time and O(N) space in this simple form; streaming counters reduce space to O(S) for S sources. Confidence bounds and population weighting need the audit design and sampling probabilities.

Common Mistakes

Do not confuse late outcomes with a true zero-error cohort.
Do not combine targeted investigations with random audits without labeling the samples.
Do not attribute every disagreement spike to the model.