Annotation prompts: define the unit and label taxonomy

Last updated: 4 Oct 202611 min read

tutorial

An annotation prompt needs a single unit of work, a finite label set, decision criteria, exclusions, and an unknown route. Ticket-level and message-level labels answer different questions. Define whether the label describes customer intent, authorized next action, or final case outcome; these cannot be used interchangeably. Each label should have observable evidence and a counterexample that separates it from its nearest neighbor. Version the taxonomy before annotators begin so later changes are traceable.

Operational case

Meridian labels one redacted ticket excerpt at a time for requested resolution: refund, replacement, or unknown. A customer who asks whether an item can be repaired is not automatically asking for a replacement. A later agent promise is not the customer's original intent. The prompt returns a label proposal with an evidence span and a scope note. If the excerpt lacks the relevant message, it returns unknown and requests the minimum earlier context.

unit: one redacted ticket excerpt
labels: REFUND | REPLACEMENT | UNKNOWN
target: customer's requested resolution
exclude: agent promise and final case outcome
output: label + evidence span + missing-context flag

Performance and review cost

Applying a stable taxonomy to N units is O(N) decisions, with O(N) stored labels and evidence spans. Ambiguous units increase review time rather than algorithmic complexity. A small taxonomy with a genuine unknown route reduces false precision without hiding which cases need a human.

Common Mistakes

Do not mix customer request with final resolution.
Do not label an entire thread when the unit is an excerpt.
Do not define categories only by their names.

Connected lessons

prompt engineering

annotation quality