Script profiles and code-switching boundaries in text intake

Last updated: 6 Oct 20265 min read

tutorial

A support message can mix Latin text, Devanagari text, product codes and emoji. Intake must preserve that mixture and avoid treating script as a language label.

Script is an observation, language is an inference

A message such as “Return ऑर्डर ZX-47” contains Latin and Devanagari letters, but neither script count nor a keyboard setting proves the writer’s language. Loanwords, transliteration and quoted product names break simple rules. Record a script profile as an intake signal only. If routing requires a language label, use reviewed data with an explicit unknown class and an abstention path. Never discard a message because a heuristic says it is outside the supported set.

Preserve identity and offsets

Store the original bytes with a decoding policy, then a decoded string used for annotation. Keep normalization choices explicit: NFC can change the number or arrangement of code points, and case folding can change string length. Do not record offsets in one representation and apply them to another. Track the transform version and keep an offset map if a normalized view is unavoidable. Unicode and tokenization and span annotations describe the linked contracts.

Find where a classifier is blind

Evaluate at message, conversation and embedded-span levels. A language detector may return English for a sentence that contains a Hindi complaint and an English product name. Segment-level detection can help analysis, but short segments are less reliable. Review error slices for Romanized Hindi, mixed numerals, script switches near entity boundaries and pasted signatures. Keep repeated conversations together in the split. Grouped temporal splits avoid a misleading score on near-duplicate messages.

Decide the handoff

A script profile can route a message to a multilingual model or a specialist review queue. It should not authorize a high-impact action. Log the chosen route, confidence band, model version and whether a human corrected it. For an unsupported mix, respond with abstention rather than force a low-confidence class. This feeds multilingual calibration and the routing project.

Implementation

import unicodedata

def script_profile(message):
    counts = {"latin": 0, "devanagari": 0, "other_letters": 0}
    for character in message:
        if not character.isalpha():
            continue
        unicode_name = unicodedata.name(character, "")
        if unicode_name.startswith("LATIN "):
            counts["latin"] += 1
        elif unicode_name.startswith("DEVANAGARI "):
            counts["devanagari"] += 1
        else:
            counts["other_letters"] += 1
    return counts

observed = script_profile("Return ऑर्डर ZX-47")
assert observed["latin"] > 0 and observed["devanagari"] > 0

Performance and operating cost

The scan is O(n) time for n Unicode code points and O(1) counters for this fixed set of script buckets. A complete script inventory would require more buckets. Unicode name lookup adds per-character overhead; cache or batch only if profiling shows intake latency matters. Script counts are cheap compared with model inference, but their uncertainty is high. Measure routing accuracy on reviewed mixed-script messages rather than trusting the counters.

Common Mistakes

Equating Devanagari with Hindi or Latin with English.
Stripping accents or non-Latin characters before annotation.
Changing normalization without regenerating offset maps.
Treating a short mixed-script fragment as a reliable language decision.