A held-out case measures behavior only if it was not repeatedly used to tune the candidate prompt. Synthetic wording does not make a case independent. The same generator can produce close siblings across training, development, and release sets; a model may pass by recognizing a familiar template rather than handling the rule. Assign each case a lineage key, mutation operator, and split before reading candidate outputs. Keep all descendants of one seed in the same split, and run near-duplicate review across splits. Lock a critical release set and record every time it is exposed during tuning. If the team changes the prompt after inspecting a holdout failure, that holdout has become development feedback; rotate or supplement it before claiming an unbiased result.
Synthetic evaluation holdouts: stop the generator from teaching the test
Operational case
The return team generates a 'missing scan with hostile note' case from seed RT-642 in the development set. A later release case changes only the item description but keeps the same note and expected answer. The new prompt passes both because its instructions were rewritten after the first case failed. The release case is not an independent test of generalization. The team groups both under one lineage, moves the sibling out of the locked set, and adds a newly reviewed case with a different evidence transition. It retains the old result for history rather than erasing the exposure.
Seed RT-642; mutation M3 -> lineage L-42.
Development: L-42/hostile-note-a.
Release draft: L-42/hostile-note-b -> reject sibling leakage.
New held-out case: different seed, different event transition.
Record every holdout exposure and prompt revision.Performance and operating cost
For C cases, lineage assignment is O(C) bookkeeping when each generated case records its parent. Cross-split near-duplicate checking can approach O(C squared) without indexing. A locked holdout is expensive to replace, so use development and adversarial suites for routine iteration and consult the release set at defined gates. Track prompt, generator, oracle, and split versions. A clean lineage lowers a known leakage risk but cannot prove that a foundation model never saw similar public text. State that uncertainty in the evaluation report rather than presenting a synthetic score as a universal capability claim.
Common Mistakes
- Do not place close seed siblings in development and release splits.
- Do not keep calling a case held out after tuning against its failure.
- Do not claim a clean split proves absence of all pretraining contamination.
Connected lessons
- Production prompt engineering
- Prompt Engineering
- Evaluation leakage: keep the release test independent
- Evaluation case ledgers: revise labels without erasing history
- Adversarial case mutations: test the boundary, not a magic phrase
- Synthetic evaluation cases: mutate a contract, not a customer's record
- Synthetic case oracles: reject invalid labels before scoring a model
- Synthetic case coverage: count distinct decisions, not rewritten sentences
- Synthetic cases are not automatically private
- Project: build a checked synthetic return-triage suite
- Synthetic evaluation-case decisions