Hosted-model dependencies: pin behavior and plan retirement

Last updated: 6 Oct 20265 min read

tutorial

A hosted model is a versioned production dependency whose availability, output contract and retirement date belong in the release manifest.

Inventory the dependency

A multilingual support-ticket router calls an externally hosted language model and expects a JSON object with queue, urgency and reason. Record deployment ID, model revision when exposed, API revision, region, prompt revision, tool schema, maximum input and output sizes, timeout, rate limit, cost unit and retirement notice. If the provider can update an alias without notice, treat the alias as mutable and monitor its resolved identity where available. Release manifests bind prompt and model; the external dependency adds lifecycle and service behavior.

Do not equate endpoint uptime with compatibility

A replacement may return HTTP success while changing field names, valid queue labels, refusal patterns, language handling or latency. Pin a consumer-owned parser schema and reject malformed outputs. Keep a fixture set of representative tickets and hard cases in several languages. Classify every response as valid route, abstention or contract failure before measuring task quality. Response contracts protect downstream consumers from shape drift.

Work backward from the retirement window

Discover announced end dates early enough to allocate evaluation, client migration and rollback time. Stand up a successor deployment while the incumbent remains available when the hosting terms allow it. Record the latest safe cutover date and what happens if no successor passes: a controlled human-routing fallback may be preferable to a silent automatic substitution. Consumer inventory identifies clients that still depend on the old queue labels.

Budget operating risk

A candidate with better average routing quality may cost more per ticket, exceed the latency budget or have a lower regional quota. Model the peak request rate and ticket-length distribution, then test the slow and long cases. Preserve approved input handling and retention rules in both deployments. Paired replay evaluates these changes before traffic shifts; the project catches a JSON shape regression hidden by a good average score.

Implementation

def dependency_ready(manifest, now_day, required_days):
    if not manifest["deployment_id"] or not manifest["api_revision"]:
        return "hold:identity"
    if manifest["retirement_day"] - now_day < required_days:
        return "urgent:migration-window"
    if not manifest["fallback_owner"]:
        return "hold:no-fallback-owner"
    return "tracked"

manifest = {"deployment_id": "ticket-route-r47", "api_revision": "r8",
            "retirement_day": 180, "fallback_owner": "support-ops"}
assert dependency_ready(manifest, 100, 47) == "tracked"
assert dependency_ready(manifest, 150, 47) == "urgent:migration-window"

Performance and operating cost

Manifest admission is O(1) time and space. Keeping two hosted deployments during migration can double reserved capacity or incur paired-request charges. Testing peak throughput and long tickets costs inference calls, but prevents an apparently compatible successor from failing under real volume.

Common Mistakes

Treating a mutable deployment alias as a stable model digest.
Checking only status codes and ignoring output meaning.
Starting the migration after the retirement window closes.
Assuming the successor has the incumbent region, quota or retention behavior.