Model Output Evaluation and Release Control

Working case

The portal team changes the model used to summarize pump inspections. In an offline review set, the candidate writes shorter summaries but misses a failed valve in 7 of 83 cases and once mixes evidence from case 63 into case 47. A pleasing average rating would hide the operational problem. The team blocks release on the cross-case leak and severe omission, traces the retrieval and permission failures separately, and replays the cases after repair. When the candidate later reaches a small cohort, the UI keeps a way to inspect the underlying notes and report a wrong statement. A kill action returns new operations to the prior route without altering already completed records.

Implementation boundary

function releaseDecision(results) {
  if (results.crossTenantLeaks > 0 || results.severeOmissions > 0) return 'block';
  if (results.checkedCases < 83) return 'collect-more';
  return 'stage';
}
console.log(releaseDecision({ crossTenantLeaks: 0, severeOmissions: 1, checkedCases: 83 }));
// Output: block

Version the prompt contract, retrieval policy, model route, output parser, and evaluation set together. Define checks for evidence coverage, unsupported claims, cross-record leakage, schema validity, empty or truncated output, latency, cost, and user correction. Use a fixed holdout set for release decisions and a separate changing set for regression exploration. Inspect failures individually; averages can mask rare high-severity events. Run generated output through a strict renderer that escapes text and validates any structured fields before use. Stage by stable cohort assignment, compare completion and correction rates with denominators, and predeclare stop thresholds. Record version IDs and operation IDs for short investigations, while keeping private content under its normal retention and access rules. Provide a narrow rollback that changes only new generation requests.

Cost and boundaries

Evaluating E cases across C candidate routes requires O(E × C) model operations plus retrieval and review time. Running every candidate on every production request can double spend, so sample intentionally and avoid sending private data twice without a product reason. A fixed test set can become stale; maintain fresh cases without tuning every change to the holdout. Human review is costly but necessary for severe error categories. Track latency percentiles, output bytes, cost units, correction rate, unsupported-claim rate, and the number of records with missing or cross-case evidence. Report both count and denominator.

Failure trace

Insert one case where the correct answer is no summary because permission was revoked. Add a retrieved note with a hostile instruction and ensure no tool side effect occurs. Truncate a stream before its terminal marker and require an incomplete outcome. Return syntactically valid structured output with a nonexistent evidence ID and reject the unsupported reference. Increase the candidate cohort, then inject a severe omission and activate the kill action; new requests must use the prior route while existing operation IDs retain their own version metadata. Compare two releases on the same 83-case holdout and inspect individual severe failures, not only the average score.

Verification

Severe errors cannot disappear inside an average.
Output references only authorized evidence IDs.
A rollback affects new operations without falsifying old status.

Practice drill

Create an 83-case release set with 12 missing-evidence cases, 9 permission-change cases, 7 hostile-note cases, and 55 ordinary cases. Define a severe-error threshold of zero for cross-tenant disclosure and a separate threshold for missing safety-critical findings. Run the candidate, inspect all violations, and document the exact retrieval revision and output version. Stage the repaired route to a small stable cohort. Trigger the stop threshold and prove the flag reverts new operations while status pages still explain completed candidate operations.

Decision note

Release quality is a collection of specific observable failures and user outcomes, with a fast rollback for new requests.

Common Mistakes

Using a single fluency score as a release gate.
Rendering model output as trusted markup.
Changing the route without retaining version identity.

Connected practice

Build Project: permission-bound maintenance summary and review Web Development: model-backed application decisions quiz.