Project: route support tickets with auditable text evaluation

Last updated: 6 Oct 20265 min read

project

Build a ticket router from frozen submission-time text, prove group and time isolation, and release it with an abstention path.

Build the input manifest

Create a governed fixture with ticket ID, conversation ID, submission time, visible text and final queue. Include edits, repeated customer text, delayed labels and at least one message in another language. State the annotation policy. The model input must include only text available at routing time. The corpus contract] is the first deliverable.

Fit a baseline safely

Split by conversation and future time, then fit a vectorizer and classifier only on the training partition. Keep one untouched final test partition. Compare majority-class, rule-based and sparse-model routes under the same cost metric. Count empty feature outputs and inspect frequent errors. The sparse baseline] makes a useful reference.

Choose an action policy

On validation data, select a threshold for manual review within a stated staffing budget. Report per-class and language-slice counts, wrong urgent routes and abstentions. Do not count an abstention as an automated success. Slice analysis] shows where the router should stop making decisions.

Exercise deployment failure

Save the fitted pipeline, class mapping and version. Reject an empty and oversized request. Smoke-test known ticket texts, then deliberately swap class order in a candidate manifest and ensure promotion fails. Submit code, data manifest, split IDs, evaluation counts and a rollback procedure.

Implementation

def check_ticket_project(training_conversation_ids, evaluation_conversation_ids, predictions, labels):
    if set(training_conversation_ids) & set(evaluation_conversation_ids):
        raise ValueError("conversation identity leaked")
    if len(predictions) != len(labels):
        raise ValueError("prediction count mismatch")
    review_count = sum(route == "manual-review" for route in predictions)
    return {"evaluated": len(labels), "manual_review": review_count}

Performance and operating cost

Vectorization and model fitting scale with nonzero text features and solver iterations. Split auditing is O(N) for exact identities; manual review capacity is a separate recurring operational cost.

Common Mistakes

Do not include agent replies written after the routing decision.
Do not report a model score without an operational baseline.
Do not deploy a vectorizer separately from its classifier.