Model settings: change one generation variable against a fixed case set

Decision in practice

A legal-operations team summarizes 42 standard lease notices. At one setting the responses are concise, but two omit the notice period. Raising output length recovers those fields but increases review time; changing sampling alone does not fix a missing jurisdiction field in the prompt. The team adds that field to the input contract and reruns the same held-out notices under two settings. It keeps the configuration with no unsupported legal conclusion and a tolerable P95 latency, while routing contradictory notices to a human.

Fixed cases: 42 notices with expected fields and uncertainty labels.
Variant A: setting set A, output cap 420.
Variant B: one changed setting, same prompt and cases.
Record: omission rate, unsupported claim count, P95 latency, cost per accepted case.

Performance and operating cost

With N cases, V variants, and R repeats, evaluation calls scale as O(NVR). Reduce spend by screening obvious failures on a development subset before running the fixed holdout. A low-variation setting can still produce errors, and identical settings may not guarantee identical outputs across service changes. Keep model version and date in the result. Do not compare two variants if a retrieval index or policy version changed between their runs.

Common Mistakes

• Do not tune by reading only one attractive response.
• Do not change the model and prompt in the same uncontrolled comparison.
• Do not treat a low temperature as a correctness guarantee.

Connected lessons

• Prompt Engineering
• Prompt patterns
• Evaluation sets: measure the failure cases that matter
• Prompt budgets: trade output quality against cost and tail latency
• Instruction conflicts: resolve authority before wording
• Streaming responses: validate the complete artifact before an effect
• Project: regression-test a customer triage prompt
• Advanced prompt engineering decisions