Calibrate multilingual text decisions and fallback routes

Last updated: 7 Oct 20265 min read

tutorial

A shared classifier score is not equally trustworthy across languages. Evaluate each reviewed slice and send uncertain messages to a defined fallback.

One threshold rarely serves every slice

A ticket router may assign the right queue to 91% of reviewed English tickets but only 62% of Romanized Hindi tickets. An aggregate score can conceal this gap if most traffic is English. Store reviewed language and script-mix slices separately from model guesses. Compare confidence against correctness within each slice, with support counts and uncertainty. Do not tune thresholds against the final test set. Classification slices gives the base measurement pattern.

Calibrate against held-out decisions

Fit a calibration map or thresholds on a validation set that is separate from training and final audit data. Group messages by conversation and time before splitting; otherwise repeated phrasing gives optimistic probabilities. Estimate coverage, selective accuracy, wrong-queue cost and human-review volume for each candidate threshold. A threshold is part of the deployed decision rule, so a threshold change requires review even when weights remain the same. Grouped temporal validation defines the separation.

Choose the safe branch

A route can be automatic only when the score and slice policy permit it. Otherwise send to a specialist queue with the original text and a reason code, such as unsupported language mix or low margin between queues. Unknown language is not the same as low confidence; a message can be confidently misclassified when the model has never seen its dialect. Add an out-of-distribution review sample and monitor correction rates after release. The intake hints in script profiling may select a review workflow, but they must not masquerade as a validated language detector.

Monitor the moving distribution

Track traffic counts, automatic coverage, error on reviewed samples and backlog by language and script slice. A sudden rise in mixed-script tickets may reflect a campaign, a product launch or a changed input channel. Sample some high-confidence auto-routes for review; otherwise you only observe labels for abstained tickets and underestimate errors. Freeze the label taxonomy and model bundle in the serving contract.

Implementation

def route_with_abstention(scores, thresholds, reviewed_slice):
    if not scores:
        return "manual-review", "no-scores"
    selected_queue = max(scores, key=scores.get)
    threshold = thresholds.get(reviewed_slice)
    if threshold is None:
        return "manual-review", "unsupported-slice"
    if scores[selected_queue] < threshold:
        return "manual-review", "below-threshold"
    return selected_queue, "automatic"

queue_scores = {"returns": 0.78, "billing": 0.13, "delivery": 0.09}
assert route_with_abstention(queue_scores, {"mixed-reviewed": 0.82}, "mixed-reviewed") == (
    "manual-review", "below-threshold"
)

Performance and operating cost

Selecting the highest score is O(k) time for k queues and O(1) extra space. Calibration is an offline fit; serving a threshold lookup is constant time. The expensive part is review capacity: expected manual cases equal arrival volume multiplied by abstention rate. Report that forecast by slice, then compare it with the observed queue. Avoid very fine slice thresholds when support is too small; use a broader fallback until more reviewed examples exist.

Common Mistakes

Treating raw softmax output as calibrated probability.
Optimizing every threshold on the final audit holdout.
Assuming a high score proves the message is in a supported language.
Measuring accuracy only on automatically accepted tickets.