Dual-label evaluation: compare models across a target migration

Last updated: 6 Oct 20265 min read

tutorial

During a target change, evaluate both the new task and the legacy consumer view without pretending their metrics are identical.

Keep two clearly named views

The new scanner model emits torn label, blurred print, other damage and clear. The old consumer still needs a binary damaged-versus-clear route. Produce the new four-class output and a reviewed projection to the legacy route. Evaluate native class performance on adjudicated new-taxonomy examples, and evaluate the binary projection on a separate legacy-compatible frame. Do not compare four-class accuracy with old binary accuracy as a before-and-after improvement. Taxonomy revisions define both meanings.

Measure migration coverage

Count records with deterministic mapping, independently relabeled records, unresolved ambiguous records and excluded data. Report quality per new class and by acquisition channel; a strong average can conceal that rare torn labels are missed. When audit labels are selected through review routing, the observed set may be biased. Independent audit coverage helps estimate errors outside the reviewed queue.

Test old clients explicitly

Run output contract tests for clients that receive only the projected binary result, and new clients that need the fine-grained class and evidence. Unknown labels, low confidence and malformed outputs must take documented fallbacks. Pin client, model, taxonomy and projection revisions in each decision. Response contracts prevent a silent shift in field meaning, while client migration tests cover mixed deployments.

Cut over only with two passing gates

Require acceptable new-class quality and stable legacy route behavior before expanding. The new model can improve fine-grained reporting yet still change the binary review route for older consumers; that is a release failure until examined. Keep the prior model and projection available through the cutover, then retire them only after consumer inventory confirms nobody needs them. Retirement gates close the migration safely; the project runs the full dual view.

Implementation

def legacy_damage_route(new_label, projection):
    if new_label not in projection:
        return "manual-review:unknown-label"
    return "review" if projection[new_label] else "release"

projection = {"clear": False, "torn-label": True,
              "blurred-print": True, "other-damage": True}
assert legacy_damage_route("torn-label", projection) == "review"
assert legacy_damage_route("clear", projection) == "release"
assert legacy_damage_route("possible-tear", projection) ==        "manual-review:unknown-label"

Performance and operating cost

Projection lookup is O(1) expected time and space. Dual evaluation runs two metric suites and may require extra adjudication and storage for newly labeled records. The cost is temporary but necessary: a single aggregate score cannot establish both new-task quality and compatibility with the old decision route.

Common Mistakes

Comparing old binary accuracy directly with new four-class accuracy.
Using ambiguous mapped records as if they were adjudicated labels.
Testing the new model without exercising older clients.
Retiring the binary projection before consumer inventory reaches zero.