Project: build a receipt-labeling and adjudication packet

Prepare evidence and rules

Create 47 synthetic receipt records across three scanner sources with clean, cropped and duplicated cases. Give each an immutable ID, evidence revision and event timestamp. Write a four-state taxonomy with inclusion and exclusion rules. The receipts are a controlled fixture for exercising the workflow, not a claim that generated examples represent live traffic. The label contract should be readable without a model score.

Assign and adjudicate

Randomly select an audit set from the full fixture and route a separate hard-case queue. Give 12 items two blind reviews by distinct reviewers. Retain each raw decision, confidence, reason and guideline version. Adjudicate conflicts with a recorded policy or evidence reason; leave truly unresolved items unknown. Compare overall and per-class agreement without treating agreement as accuracy.

Freeze and evaluate

Version the adjudicated gold set. Hold out a source and later time period for a simple classifier or rule system; duplicate scans must stay in one split. Report coverage, unknown rate and errors by scanner. Revise three labels under a new guideline and show the exact gold-set diff. A frozen reference makes the comparison reproducible.

Deliver the audit trail

Submit the fixture generator, rules, assignment ledger, original labels, adjudication log, gold versions, evaluation script and a one-page error review. Include one item that changed because the evidence revision changed and another that changed because the policy changed. State which sample supports a population estimate and which sample was only used to improve the model.

Implementation

def review_packet_complete(packet):
    required = {"guideline_version", "assignment_ledger", "raw_reviews",
                "adjudications", "gold_version", "split_manifest"}
    return required <= packet.keys() and all(packet[field] for field in required)

Performance and operating cost

Packet field validation is O(1) for a fixed contract. Generating and checking R reviews costs O(R); adjudication work is driven by disagreements and policy gaps, while duplicate-aware splitting requires a group index.

Common Mistakes

Do not reuse the hard-case queue as an unbiased final test.
Do not discard pre-adjudication reviewer decisions.
Do not report a changed score without the gold-set version.