Project: regression-test a customer triage prompt

Build the candidate

Define labels urgent, routine, and review, then name the evidence each label requires. Normalize service identifiers and timestamps before the model call. Render case text through a bounded data slot and keep policy instructions in a versioned template. Produce a structured response with decision, evidence IDs, and uncertainty reason. Validate the final object in application code. A streaming preview may be displayed as a draft, but no ticket-routing effect is allowed until completion and validation.

Run the regression suite

Reserve 26 cases as a holdout. Create paired variants by changing irrelevant customer tokens and wording while preserving the hazard. Add material variants by removing a required policy clause or changing a case timestamp. Compare decision parity, unsupported claims, referral rate, and P95 latency. Repeat a subset under two model settings, changing only one setting at a time. Record the template hash, model ID, retrieval version, and case IDs; investigate every high-consequence mismatch before release.

Packet: 74 fictional cases; 26 untouched holdout cases.
Invariant: equivalent hazard -> same urgency and evidence IDs.
Material change: required policy clause absent -> review.
Stop: any unauthorized effect or unsupported urgent decision.
Rollback: restore the prior prompt, schema, and retrieval bundle.

Performance and operating cost

If the suite has N cases, V variants, and R repeats, model calls scale as O(NVR). Limit early experiments to a development slice, but run the frozen holdout before a canary. Keep the packet small enough for a reviewer to inspect and record raw critical-error counts, not only an average grade. A cache may reduce repeated policy input cost, provided its tenant scope and policy version are correct. The final packet includes failures, effect logs, and a human-review route for unresolved cases.

Common Mistakes

• Do not tune the prompt on the untouched holdout.
• Do not count a partial stream as a completed routing decision.
• Do not compare variants while policy and model settings also change.

Connected lessons

• Prompt templates: bind variables without changing instruction structure
• Streaming responses: validate the complete artifact before an effect
• Metamorphic tests: verify behavior when harmless details change
• Fairness checks: test equivalent cases across groups and wording
• Prompt traces: connect an answer to its inputs, checks, and effects
• Model migration: replay contracts before changing providers or versions
• Advanced prompt engineering decisions