Annotator assignment: coverage, blinding and independent review

Last updated: 5 Oct 20265 min read

tutorial

Assignment rules decide which items receive independent judgment and whether reviewers are exposed to the same bias.

Sample before routing

Select an audit sample from the incoming population before a model confidence filter or a reviewer queue changes it. Stratify by document source, image quality and expected rarity, and retain selection probabilities. A queue containing only low-confidence receipts is useful for fixing difficult cases, but cannot estimate overall label accuracy. Population and sampling frame define what the audit can claim.

Keep independent passes independent

Assign two reviewers to the same item without showing the first answer to the second. Rotate reviewer pairs so one specialist does not monopolize a source. Record assignment time, reviewer role, guideline version and whether an AI suggestion was displayed. If assisted reviews are operationally necessary, reserve an independent blind set to measure suggestion-induced agreement.

Protect access and capacity

A reviewer should see the least data needed for the question. Remove account identifiers unless they determine the label, and use stable pseudonymous item IDs for joins. Budget enough repeated items to estimate reviewer consistency, but not so many that repeat recognition dominates. A capacity cap by reviewer prevents a single person’s interpretation from defining the apparent ground truth.

Exercise assignment fairness

Suppose 47 receipts arrive from three scanners and 12 require duplicate review. Pick duplicate reviews across all scanner sources and image-quality bands. Check that no reviewer is assigned both independent passes for one receipt. Then add a specialist-only scanner and state that its quality estimate cannot be generalized to other scanners without overlap or a separate sample.

Implementation

def independent_pairs(assignments):
    by_receipt = {}
    for assignment in assignments:
        by_receipt.setdefault(assignment["receipt_id"], set()).add(assignment["reviewer_id"])
    return {receipt_id for receipt_id, reviewers in by_receipt.items()
            if len(reviewers) >= 2}

Performance and operating cost

Grouping A assignments costs O(A) expected time and O(R) space for R distinct receipts. Balanced scheduling needs source and reviewer-capacity indexes; the audit sample must preserve its original selection probabilities.

Common Mistakes

Do not call model-filtered queues representative audits.
Do not expose the first judgment during an independent second pass.
Do not ignore source coverage when assigning specialist reviewers.