Evaluation sets: measure the failure cases that matter

Last updated: 5 Oct 20268 min read

tutorial

A useful evaluation set groups cases by decision type, evidence quality, language, document size, and consequence of error. Keep a stable holdout that prompt authors do not tune against directly; use a separate development set for iteration. Define the grading rule before inspecting model output. Binary pass rules work for schema, prohibited effects, and evidence references, while editorial quality needs a rubric and human review. Track both aggregate performance and the worst important slice. A prompt that improves common cases while worsening abstention may be a regression.

Decision in practice

A claims team samples 73 records: clean approvals, expired coverage, missing receipts, contradictory dates, and scanned forms with weak OCR. It labels each record with the accepted decision and acceptable uncertainty. When a new prompt scores better overall but turns two missing-receipt cases into approvals, the release gate fails. The team keeps those cases in the holdout and adds a fresh development case for the new failure pattern. Evaluators see record IDs and policy versions, so a later policy update can be separated from prompt drift.

Slices: clean, missing evidence, conflict, OCR uncertainty, unauthorized action.
Gate: zero unauthorized effects; no regression on missing-evidence decisions.
Record: prompt version, model version, case ID, verdict, reason.

Performance and operating cost

Evaluation cost grows with case count and repeated model runs. If N cases are each run R times, model calls are O(NR); judge calls add another multiplier. Sample where uncertainty is high, but retain a fixed core for comparison. Report confidence intervals or raw counts when slices are small, and avoid claiming that a change from one error to zero proves general reliability. A human adjudication queue should capture disagreements rather than silently changing expected labels.