Multi-passage answers: claim alignment and conflicting evidence

Last updated: 6 Oct 20265 min read

tutorial

Several passages can support different parts of an answer or disagree. Align claims by entity, action and revision before combining them.

Make each passage a bounded claim

A rollback section says to disable retries before redeployment; a later verification section says to watch error rate for 47 minutes. Extract each statement with its service, action, condition and source span. Do not merge passages because they share a keyword. A note about staging may not apply to production. Predicate roles and negation help keep “disable retries” separate from “retries were not disabled.”

Align revisions and applicability

A new appendix can override an older procedure, but its existence does not automatically invalidate every prior step. Record document revision, effective date, environment and explicit supersession relation. If two active passages prescribe different actions under the same conditions, report a conflict for review. Do not quietly choose the later timestamp without checking authority and scope. Claim conflict review provides the broader contradiction policy.

Require evidence for every answer part

A final answer may have three sentences drawn from three passages. Keep a mapping from each sentence to the passage or table cell that supports it. If a sentence combines conditions from two sources, both must be listed, and the combination itself needs review. A retrieved passage that merely mentions the topic is not support for a specific instruction. Summary claim verification can then test whether the wording exceeds the evidence.

Measure unsupported synthesis

Evaluate correct action, condition, environment, evidence span and conflict detection on a frozen incident-scoped holdout. Include distractor passages with the same service but different revisions. Count answers that omit a necessary prerequisite or present a disputed step as settled. Reviewer time matters: if every answer needs full re-reading, the system has not reduced the operator burden. The project checks this at the answer level.

Implementation

def active_claim_values(claims, service, environment, revision):
    relevant = [claim for claim in claims
                if claim["service"] == service and
                   claim["environment"] == environment and
                   claim["source_revision"] == revision]
    values = {claim["instruction"] for claim in relevant}
    if len(values) > 1:
        return {"state": "review", "reason": "conflicting-instructions",
                "evidence": [claim["passage_id"] for claim in relevant]}
    if not values:
        return {"state": "unanswered", "evidence": []}
    return {"state": "supported", "instruction": next(iter(values)),
            "evidence": [claim["passage_id"] for claim in relevant]}

claims = [{"service": "gateway-west", "environment": "production",
           "source_revision": "r5", "instruction": "disable retries",
           "passage_id": "rollback-47"}]
assert active_claim_values(claims, "gateway-west", "production", "r5")["state"] == "supported"

Performance and operating cost

Scanning c claims is O(c) time and O(c) space for the filtered records and evidence IDs. Indexed service and revision fields can narrow the scan. This code detects conflicting instruction strings, not semantic contradictions; reviewers still need to compare conditions, authority and wording before a conflict is resolved or a procedure is released.

Common Mistakes

Combining staging and production instructions into one answer.
Choosing a newer passage without checking whether it supersedes the older one.
Using a topical mention as evidence for a precise action.
Failing to expose a conflict between two active instructions.