Prediction-set monitoring: coverage lag, set width and review load

Last updated: 7 Oct 20265 min read

tutorial

Monitor prediction sets with mature outcomes and an explicit review-capacity budget; unlabeled width is only an early signal.

Separate early and late measurements

The scanner can report set size immediately, but true coverage is known only after an inspected parcel receives a resolved damage label. Record predicted set, model and cutoff revisions, cohort, event time and label maturity. Report set width and manual-review rate daily; report coverage only on mature joined outcomes. Do not code missing labels as misses or successes. Mature outcome joins establish a comparable denominator.

Watch selective labels

Parcels with wide sets are more likely to be sent to review, so the labeled cohort overrepresents uncertain cases. A coverage estimate from that queue does not describe all auto-cleared parcels. Reserve a small independent audit sample from cleared traffic and report it separately. Show counts and intervals by camera and package class. A nominal target does not save a sparse slice. Audit sampling provides a frame outside the review policy.

Bound operational cost

A prediction set is only useful if its route can be handled. Define how singletons, multiple classes and full sets map to automated clearance or human inspection. Track queue age and reviewer capacity; a new cutoff that increases coverage by sending every parcel to manual review fails the workflow. Evaluate both the incumbent and candidate on the same frozen traffic frame. Threshold policy changes also alter reviewer load and need the same capacity discipline.

Respond to shift without claiming guarantees

A new scanner lens or packaging material can break the similarity between calibration and production populations. Rising set width is an early signal, not proof that coverage fell. When mature audit labels show low coverage or a new cohort lacks evidence, pause automated use for that cohort, investigate data quality, then recalibrate on an approved new frame and requalify the release. Drift monitoring and the project connect diagnosis to reversible action.

Implementation

def coverage_report(decisions, minimum_mature):
    mature = [row for row in decisions if row["outcome_mature"]]
    if len(mature) < minimum_mature:
        return {"state": "insufficient", "mature": len(mature)}
    covered = sum(row["true_class"] in row["prediction_set"]
                  for row in mature)
    mean_width = sum(len(row["prediction_set"]) for row in decisions) / len(decisions)
    return {"state": "measured", "coverage": covered / len(mature),
            "mean_width": mean_width, "mature": len(mature)}

rows = [{"prediction_set": {"torn", "wet"}, "true_class": "torn",
         "outcome_mature": True},
        {"prediction_set": {"clear"}, "true_class": "wet",
         "outcome_mature": True},
        {"prediction_set": {"crushed", "wet"}, "true_class": None,
         "outcome_mature": False}]
report = coverage_report(rows, 2)
assert report["coverage"] == 0.5 and report["mean_width"] == 5 / 3

Performance and operating cost

Aggregation scans n decisions in O(n) time and O(n) temporary space in this sample; streaming counters use O(1) state per cohort. Independent audit labels and human review consume capacity. Recalibration adds a new fit and evaluation cycle, while merely raising the cutoff can hide a weak model behind wider sets.

Common Mistakes

Treating width as measured coverage before labels arrive.
Reporting review-queue coverage as coverage for all traffic.
Ignoring a multi-class-set surge that overwhelms reviewers.
Assuming an old coverage target survives a population shift.