Skip to content
AITroveRead. Build. Understand.
Reading list
Make this comfortable

Morphology ambiguity, index versions and rollback

Last updated: 7 Oct 20265 min read
tutorial
AdvancedBy AITrove Editorial

A new lemma map changes which documents a query can reach. Ship it as an index version with an audit and rollback path.

Record the transform chain

The searchable form depends on decoding, token boundaries, language choice, part-of-speech tagging, lemma rules and protected-term policy. Record a digest for that bundle with each derived index. If one component changes, build a new index namespace or rebuild all affected postings. Mixing query forms from a new analyzer with old document postings causes silent misses. Embedding version pairing follows the same principle for vector indices.

Inspect ambiguous readings

Sample short support headings, commands, proper nouns and code-switched messages. The word “left” can describe direction or a completed departure; a tagger with no context may choose incorrectly. Permit an abstain state that indexes the surface form only. Review false expansions that pull unrelated cases into top results. If a query contains an exact error code, preserve its exact-match channel even when other words are expanded.

Reindex without breaking search

Build the new postings from immutable source revisions and compare them with the current index. Check document counts, tenant scope, query parity on exact identifiers and ranking metrics on reviewed queries. Move a search alias only after the audit passes. Keep the old index until rollback is no longer required. A deleted source must be excluded from both versions; retaining it in a shadow index is a privacy failure.

Measure the release

Report unmatched exact queries, recall at a chosen depth, false positives caused by expansion, p95 latency and index size. Separate new-language and technical-term slices. The term contract defines what may be expanded. The support search project turns these checks into an operational gate.

Implementation

python
def eligible_for_index_swap(old_manifest, new_manifest, audit):
    if old_manifest["tenant_scope"] != new_manifest["tenant_scope"]:
        return False
    if new_manifest["source_revision"] != audit["source_revision"]:
        return False
    return (audit["exact_code_misses"] == 0
            and audit["deleted_source_hits"] == 0
            and audit["recall_at_8"] >= audit["minimum_recall_at_8"])

old = {"tenant_scope": "support-eu", "source_revision": "r46"}
new = {"tenant_scope": "support-eu", "source_revision": "r47"}
audit = {"source_revision": "r47", "exact_code_misses": 0,
         "deleted_source_hits": 0, "recall_at_8": 0.84,
         "minimum_recall_at_8": 0.81}
assert eligible_for_index_swap(old, new, audit)

Performance and operating cost

The release gate is O(1), while a full reindex is O(t) tokenization work plus postings writes for t source tokens. Running two indices temporarily increases storage and refresh cost. Treat that cost as part of a reversible release, and measure the shadow index for deletion compliance before moving the alias.

Common Mistakes

  • Deploying a query analyzer before rebuilding document postings.
  • Keeping removed records in the rollback index.
  • Scoring only average queries while exact identifiers fail.
  • Using a language-specific lemma map after uncertain language detection.

Read next

ai-data
natural-language-processing
Storage details