Generative releases: bind prompt, model, tools and output contract

Last updated: 7 Oct 20265 min read

tutorial

A generative feature changes when any prompt, model, retrieval index, tool schema or decoding policy changes.

Version the complete behavior

A maintenance-ticket summarizer uses a prompt template, hosted language model, retrieved manual passages, a ticket lookup tool and a JSON output contract. A prompt edit is a behavior change even if the model name stays fixed. Record immutable revisions for the template, model, retrieval index, tool schema, output validator, decoding settings and fallback route. A provider alias may shift without a visible name change, so record the resolved model identity when available. Chain identity is the operational base for this manifest.

Keep inputs and authority separate

Ticket text and retrieved manuals are task data, not policy instructions. The application should constrain tool access outside the prompt, validate requested tool arguments and prevent a generated sentence from authorizing actions. The release packet names permitted data scopes and any redaction step. Test malicious or malformed retrieved passages as inputs; a successful output-format check cannot prove the content is trustworthy. Logging privacy limits retention of sensitive ticket text and generated responses.

Use a stable response contract

Define required fields, allowed enum values, maximum lengths and what happens when generation is incomplete or invalid. Validate output after the model returns; do not assume a prompt instruction guarantees JSON or correct field semantics. Keep an explicit abstain or human-review state for unsupported claims. A retry should preserve the same release manifest and idempotency key. Request contracts protect callers when the output schema changes.

Promote and roll back as a bundle

Evaluate the whole manifest, then stage a cohort whose outputs are observable under privacy rules. A rollback must restore prompt, model selection, retrieval index and tool contract together, or the prior prompt may query an incompatible tool. Evaluation gates test quality and failure behavior, while the project catches a stale retrieval index after a prompt update.

Implementation

def generative_manifest_ready(release, available):
    required = {"prompt_revision", "model_revision", "index_revision",
                "tool_schema_revision", "output_schema_revision"}
    if set(release) != required or any(not release[name] for name in required):
        return "hold:incomplete-manifest"
    for name in required:
        if release[name] not in available[name]:
            return "hold:missing-" + name
    return "stage:bundle"

release = {"prompt_revision": "summary-p47", "model_revision": "text-r8",
           "index_revision": "manuals-i31", "tool_schema_revision": "ticket-t4",
           "output_schema_revision": "summary-s3"}
available = {name: {value} for name, value in release.items()}
assert generative_manifest_ready(release, available) == "stage:bundle"
assert generative_manifest_ready({**release, "index_revision": "manuals-i30"},
                                 available) == "hold:missing-index_revision"

Performance and operating cost

Manifest validation is O(k) expected time and O(k) temporary space for k components. Evaluation and staged serving cost model tokens, retrieval queries, tool calls and human review. Treating every component as a versioned release increases bookkeeping but prevents a prompt-only rollback from leaving an incompatible index or tool contract live.

Common Mistakes

Pinning the model while leaving the prompt and index mutable.
Treating retrieved text as instructions with the same authority as application policy.
Assuming prompted JSON removes the need for output validation.
Rolling back the prompt without restoring its tool and retrieval dependencies.