Project: answer a local-language question from a foreign-language runbook

Last updated: 7 Oct 20265 min read

project

Retrieve an authorized runbook passage across languages, verify its original wording and return an answer only when the evidence supports it.

Define the case

A Hindi-speaking operator asks for the rollback limit of gateway-west. The current runbook passage is in English; an older translated note has a different threshold. Keep query and answer language explicit, preserve the service ID and index the runbook by passage revision. The test includes a code-switched question, a restricted appendix and an unrelated gateway-east result. The request contract prevents protected terms from disappearing in a translated candidate.

Build judged evidence

Annotators mark relevant source spans and exact revision IDs before seeing the system output. Include paraphrased English passages, mixed-script ticket summaries and a passage that quotes an obsolete limit. Group translations and copies by source family so evaluation does not treat them as independent documents. Freeze the judgment set by language pair and service. A reviewer must distinguish relevance to the question from support for a particular numeric answer.

Stage search and verification

Generate candidates using the raw query, a reviewed translation and a multilingual representation. Enforce access policy before ranking. Deduplicate by canonical passage ID, record which route found each candidate and verify the top evidence against its source-language text. If the source language cannot be verified or the passage is stale, answer with a visible unresolved state. Ranking review separates candidate recall from claim support.

Gate the release

Report recall, first relevant rank, answer support, identifier preservation, stale-passage errors and abstention by language pair. Test the old note and current passage in the same fixture. Release only when a supported answer traces to an authorized current span. The code below checks the final evidence contract; it does not pretend to translate text or decide relevance. Locale evaluation covers the wording of the final response.

Implementation

def answer_from_verified_passage(candidate, requested_service,
                                 active_revision, allowed_access):
    if candidate["access_level"] not in allowed_access:
        return {"state": "unanswered", "reason": "access"}
    if candidate["service_id"] != requested_service:
        return {"state": "unanswered", "reason": "service-mismatch"}
    if candidate["revision_id"] != active_revision or not candidate["verified"]:
        return {"state": "unanswered", "reason": "unverified-source"}
    return {"state": "supported", "source_passage_id": candidate["passage_id"],
            "source_language": candidate["language"],
            "answer_value": candidate["reviewed_answer_value"]}

passage = {"passage_id": "runbook-west-82", "language": "en",
           "access_level": "operations", "service_id": "gateway-west",
           "revision_id": "runbook-r8", "verified": True,
           "reviewed_answer_value": "82 minutes"}
assert answer_from_verified_passage(passage, "gateway-west", "runbook-r8",
                                    {"operations"})["answer_value"] == "82 minutes"
assert answer_from_verified_passage(passage, "gateway-east", "runbook-r8",
                                    {"operations"})["state"] == "unanswered"

Performance and operating cost

The final evidence check is O(1) expected time and space for fixed-size fields and an access set. End-to-end cost includes translation or embedding, candidate retrieval, ranking and human source review. A verified passage does not automatically make every paraphrase of its contents correct; the localized answer still needs claim-level checking.

Common Mistakes

Answering from a stale translated note when the source runbook changed.
Returning a restricted appendix because it ranked first.
Inferring an answer from a passage that merely mentions the service.
Hiding uncertainty when no reviewer can verify the source language.