Text inference: package tokenizer, labels and reject paths

Last updated: 5 Oct 20265 min read

tutorial

A serving endpoint must use the fitted preprocessing artifact, stable class order and bounded request handling from the evaluated model.

Bind the artifacts

A model file alone does not define text behavior. Package normalization, tokenizer or vectorizer, class mapping and threshold with the weights. Verify their versions at startup. Rebuilding a vocabulary from incoming tickets changes feature positions and invalidates the classifier. Sparse baselines] should serve through the fitted pipeline, not a second preprocessing script.

Bound inputs

Reject empty, undecodable and oversized messages before allocating large intermediate arrays. Set policy for attachments and unsupported languages. Preserve original ticket ID for idempotent routing requests, but keep customer text out of routine application logs. An inference response should identify model version, predicted queue and whether it abstained.

Measure both speed and meaning

Load the artifact once per worker, warm it and measure p95 latency with realistic text lengths and concurrency. Track rejection, abstention and queue distributions. A sudden shift may be a product launch, a source parser error or model behavior change; investigate before retraining. Monitoring] separates these signals.

Test a rollback

Run known-good messages through candidate and current models. Confirm class names, preprocessing hash and accepted size range. Keep the previous complete artifact available. A deployment can pass health checks while swapping two class IDs, so include expected route outputs in the smoke set.

Implementation

def route_ticket(request, loaded_pipeline, class_names, model_version):
    if not request.ticket_id or not isinstance(request.text, str) or not request.text.strip():
        raise ValueError("ticket ID and text required")
    if len(request.text.encode("utf-8")) > 180_000:
        raise ValueError("ticket text exceeds byte limit")
    predicted_queue = loaded_pipeline.predict([request.text])[0]
    if predicted_queue not in class_names:
        raise RuntimeError("model returned unknown queue")
    return {"ticket_id": request.ticket_id, "queue": predicted_queue,
            "model_version": model_version}

Performance and operating cost

Sparse inference visits input tokens and nonzero features; latency also includes decoding, queue wait and serialization. A text-length cap bounds worst-case memory and processing time.

Common Mistakes

Do not rebuild vocabulary at serving time.
Do not omit class mapping from the deployed artifact.
Do not log raw customer text by default.