Prompt releases: version the whole decision path and keep a rollback

Decision in practice

A service desk changes the response contract for ticket triage. Version PE-28 introduces an explicit unknown status and a new retrieval filter. The canary receives 7% of routine tickets; urgent security tickets remain on the established route until the failure gates pass. The team watches parser failures, unsupported citations, review referrals, and median and tail latency. A spike in parser failures triggers a rollback to PE-27 with its matching schema and retriever settings. Operators can replay a stored, redacted ticket against either version.

Release bundle PE-28: prompt hash, model setting, schema v4, retriever filter v6.
Canary: 7% eligible routine tickets.
Stop: parser or critical-decision gate fails.
Rollback: restore full PE-27 bundle and record affected request IDs.

Performance and operating cost

Canary routing and dual evaluation add operational cost and require enough traffic to see rare failures. If a dependency changes during the test, the comparison may no longer isolate the prompt. A short replay set catches obvious regressions before live traffic, while live telemetry reveals distribution shifts. Keep version IDs in logs and outputs so an incident can be traced without storing complete private prompts indefinitely.

Common Mistakes

• Do not version only the prompt while schemas and retrieval change.
• Do not run a canary without a stop rule.
• Do not claim rollback succeeded until output and effects are checked.

Connected lessons

• Prompt Engineering
• Production prompt engineering
• Evaluation sets: measure the failure cases that matter
• Prompt budgets: trade output quality against cost and tail latency
• Model judges: calibrate rubrics and swap candidate order
• Project: defend a retrieval and action workflow
• Prompt production decisions

Further prompt decisions

Test the task on changed inputs and record what the application actually accepted or rejected.

• Prompt traces: connect an answer to its inputs, checks, and effects
• Model migration: replay contracts before changing providers or versions

Try the executable check: Code lab: block a critical prompt regression.

Related implementation

• Progressive delivery: canary checks and rollback

Continue with: Online prompt experiments: define exposure and stop rules first.