Multi-step evidence chains: join identity, revision and access

Last updated: 6 Oct 20265 min read

tutorial

Several individually relevant passages form an answer only when their entities, scope and source revisions connect correctly.

Treat the path as a typed join

A log passage can identify gateway-west as the failed service. A runbook passage can give gateway-west a rollback rule. The shared service ID is the join key; a similar service name or translated label is not enough. Record the entity type, normalized ID, source span and revision on each edge. Entity-link versioning protects the join if aliases or service ownership change.

Preserve scope across hops

An incident may be in production while a runbook passage applies only to staging. A correct service join can still yield a wrong answer when environment, region, time or access policy differs. Validate these fields at each transition. A passage from a restricted appendix cannot be used to compose a public answer simply because the first passage was public. Access filtering belongs before evidence assembly as well as before final display.

Avoid path laundering

Two passages may each look credible while the connection between them is inferred without support. Store the exact edge claim and evidence span that justifies it. If a link depends on a guessed alias or outdated revision, mark the chain unresolved. Do not let answer generation smooth over a missing middle step. Unanswered states are necessary when the available passages do not connect.

Evaluate entire chains

Measure passage recall, edge correctness, complete-chain support and final answer accuracy. Include distractor chains that share one entity but differ on environment or revision. A metric based only on whether the final number matches may reward a lucky guess or memorized rule. The incident project requires a traceable path from event to service to current rule.

Implementation

def join_incident_to_rule(event_fact, rule_fact, active_revision,
                          allowed_access):
    if event_fact["service_id"] != rule_fact["service_id"]:
        return {"state": "unresolved", "reason": "service-join"}
    if event_fact["environment"] != rule_fact["environment"]:
        return {"state": "unresolved", "reason": "environment-scope"}
    if rule_fact["revision_id"] != active_revision:
        return {"state": "unresolved", "reason": "stale-rule"}
    if rule_fact["access_level"] not in allowed_access:
        return {"state": "unresolved", "reason": "access"}
    return {"state": "supported", "service_id": event_fact["service_id"],
            "rule_passage_id": rule_fact["passage_id"]}

event = {"service_id": "gateway-west", "environment": "production"}
rule = {"service_id": "gateway-west", "environment": "production",
        "revision_id": "r8", "access_level": "operations",
        "passage_id": "runbook-82"}
assert join_incident_to_rule(event, rule, "r8", {"operations"}) == {
    "state": "supported", "service_id": "gateway-west",
    "rule_passage_id": "runbook-82"}
assert join_incident_to_rule(event, {**rule, "environment": "staging"},
                             "r8", {"operations"})["state"] == "unresolved"

Performance and operating cost

Checking one fixed two-hop join takes O(1) expected time and space. For many candidate passages, pairwise joins can be O(a × b); indexing by typed entity and scope reduces candidate work. This gate checks recorded metadata, not whether either passage was extracted correctly or whether the rule text truly supports an answer.

Common Mistakes

Joining similar names without a canonical service ID.
Combining a production event with a staging-only rule.
Using one restricted hop in a public answer.
Scoring only the final answer while ignoring a broken evidence edge.