Project: release review for invoice-label acquisition

Lock the problem and pool

The classifier decides which new invoices need specialist tax-code review. Freeze policy version, supplier-grouped later test, eligible unlabeled pool and each round’s exclusion list. No test invoice, duplicate or restricted document may enter a query batch. Pool snapshots make this auditable.

Run two acquisition lanes

Start both candidates with the same initial labeled set. Spend matched analyst minutes on an uncertainty-plus-template-diversity policy and on random eligible invoices. Keep model architecture and development tuning identical. Log scores, acquisition probability where relevant, expected minutes and actual minutes. Batch design defines the costs.

Adjudicate the difficult cases

Assign independent second review to conflicts and unanswerable scans. Record first vote, second vote, evidence gap, policy escalation and final disposition. Do not collapse unknown into clear. Report the fraction of acquired records that could not be resolved under the frozen policy. Disagreement handling keeps the label stream credible.

Compare value at equal cost

At predetermined budget checkpoints, compare missed tax-code exceptions and review volume on the same untouched test. Include supplier, template and scan-quality slices with support counts. Stop if gains disappear, annotation disputes dominate or staff capacity is exceeded. Evaluation and stopping set the decision.

Make a bounded claim

The sample gate below holds an active acquisition policy even though its selected-model score improved: the comparison spent more analyst minutes than the random lane. Rebalance the budget and rerun a paired checkpoint before saying it saves labels or staff time.

Implementation

release_packet = {
    "pool_and_test_disjoint": True,
    "policy_version_locked": True,
    "active_analyst_minutes": 246,
    "random_analyst_minutes": 219,
    "allowed_budget_gap_minutes": 12,
    "active_missed_exceptions": 8,
    "random_missed_exceptions": 11,
    "unresolved_disputes": 0,
}

def review_acquisition_release(packet):
    blockers = []
    if not packet["pool_and_test_disjoint"] or not packet["policy_version_locked"]:
        blockers.append("pool or label contract failed")
    if packet["active_analyst_minutes"] > packet["random_analyst_minutes"] + packet["allowed_budget_gap_minutes"]:
        blockers.append("annotation budgets not comparable")
    if packet["active_missed_exceptions"] > packet["random_missed_exceptions"]:
        blockers.append("no exception-capture gain")
    if packet["unresolved_disputes"]:
        blockers.append("unresolved labels remain")
    return {"release": not blockers, "blockers": blockers}

decision = review_acquisition_release(release_packet)
assert decision == {"release": False,
                    "blockers": ["annotation budgets not comparable"]}

Performance and operating cost

The packet gate is O(1). The project cost includes scoring the full pool, duplicate screening, analyst minutes, independent adjudication, repeated training and a separate random-acquisition control. A claimed saving is incomplete if it counts model queries but omits staff time.

Common Mistakes

• Do not claim label efficiency from unequal annotation budgets.
• Do not include selected training cases in the final-test denominator.
• Do not approve unresolved ambiguous labels merely to complete a batch.

Read next

• Active-learning pool eligibility and round snapshots
• Active-learning batch selection under annotation cost
• Active-learning annotator disagreement and adjudication
• Active-learning evaluation and stopping by value
• Active learning with uncertainty and diversity