Annotation prompts: adjudicate conflicts and version the rule

Last updated: 7 Oct 202611 min read

tutorial

Adjudication is a recorded decision by an owner using the task definition, relevant context, and a rubric version. The record must preserve both initial labels, the evidence considered, final outcome or unknown state, rationale, and whether the rubric changed. If guidance changes, flag older items that might be affected and rerun a bounded review. A prompt can draft a conflict summary; it should not erase minority reasoning or silently convert unresolved cases to the dominant category.

Operational case

Meridian investigates seven conflicts. Three are linked to missing context and four to wording that permits two interpretations. After scoped context and owner review, five receive final labels; two stay unknown because evidence cannot distinguish outcomes. The ledger therefore has 45 labeled items, counting 40 initial agreements and five resolved conflicts. The two unknowns are excluded from label-dependent scoring under the declared policy, while their frequency remains visible.

first-pass: 40 agreed + 7 disputed = 47
resolved disputes: 5
unresolved UNKNOWN: 2
final labeled: 40 + 5 = 45
record: initial labels, context receipt, owner, rule version

Performance and review cost

Adjudicating D conflicts costs O(D) ledger visits plus human investigation. If a rule revision touches A prior items, re-review adds O(A) work. Preserving the old labels and rule versions costs storage but allows teams to explain why a metric changed after a guideline edit.

Common Mistakes

Do not delete dissenting labels after a final decision.
Do not treat unknown as a failure to be filled by guesswork.
Do not compare label counts across rubric versions without a migration rule.

Connected lessons

prompt engineering

annotation quality