Project: migrate a multilingual ticket router before model retirement

Last updated: 6 Oct 20265 min read

project

Inventory a hosted model, run paired multilingual evaluation and cut over with a tested human-routing fallback.

Register incumbent and candidate

A ticket router receives English, Hindi and Spanish support requests and emits queue, urgency and reason. Record incumbent and successor deployment IDs, prompt and parser revisions, region, quota, cost unit and retirement window. Freeze a redacted paired ticket set with adjudicated queues and urgent cases. The manifest keeps the two deployments distinct even if an API alias points to both at different times.

Catch a hidden regression

The candidate improves aggregate routing accuracy yet sometimes replaces the required urgency field with priority. Its parser rejects those outputs. After a prompt repair, the candidate still sends one urgent Spanish safety ticket to a routine queue. Hold the release; neither a success status nor a better global average justifies the error. Keep schema failures in the denominator and rerun the protected slice. Paired gates define what must pass.

Plan the actual cutover

The incumbent retires in 47 days. Reserve a shadow window, a supported-cohort canary, a soak period and a decision deadline that leaves time for fallback training. Point each accepted ticket at an explicit deployment revision and retain enough incumbent capacity for rollback until it truly disappears. If the candidate misses the deadline, route supported tickets to a staffed triage queue with queue-age monitoring rather than silently substituting an untested model.

Deliver the migration packet

Report per-language route quality, urgent misroutes, malformed outputs, abstentions, p95 latency, peak request capacity, unit cost, revision identity and remaining retirement days. Record who can switch the pointer and who owns the staffed fallback. After a successful soak, update consumer inventory and archive the old route contract. Link the final decision to retirement gates and client contract tests.

Implementation

def ticket_migration_action(candidate, days_left, minimum_window):
    if candidate["schema_failures"] or candidate["urgent_misroutes"]:
        return "hold:candidate-quality"
    if days_left < minimum_window and not candidate["fallback_staffed"]:
        return "hold:no-retirement-fallback"
    if candidate["peak_rps"] < candidate["required_rps"]:
        return "hold:capacity"
    return "canary:explicit-deployment"

candidate = {"schema_failures": 0, "urgent_misroutes": 1,
             "fallback_staffed": True, "peak_rps": 63, "required_rps": 47}
assert ticket_migration_action(candidate, 47, 21) == "hold:candidate-quality"
assert ticket_migration_action({**candidate, "urgent_misroutes": 0},
                               47, 21) == "canary:explicit-deployment"

Performance and operating cost

The release decision is O(1) time and space. Paired evaluation uses two paid inference calls per sampled ticket, with extra human adjudication for disputed routes. Staffing a fallback costs more than automatic routing, but an untested forced upgrade can create incorrect urgent decisions and an uncontrolled incident.

Common Mistakes

Accepting a candidate because its global score improved despite urgent misroutes.
Ignoring a renamed response field that breaks the consumer parser.
Waiting until the incumbent is gone to design a fallback.
Failing to record which hosted revision routed each customer ticket.