Project: audit references across incident updates

Last updated: 7 Oct 20265 min read

project

Build reviewable mention chains across incident notes, including plural references and related entities, without crossing access or revision boundaries.

Set the incident task

A responder reads updates saying “the API and worker restarted,” “they recovered,” and “the gateway check still failed.” The system proposes entity chains, a group reference and a related check, then drafts a handoff. Every displayed claim needs an active source span and reader access. Start from exact mention spans, document revisions and incident IDs rather than replacing source sentences with a model paraphrase. Group and bridging rules determine which links are identity and which are association.

Build difficult cases

Collect parallel incidents with repeated service names, staging and production updates, corrections, quotations and notes with several plausible antecedents. Annotate group membership, bridging relations, unresolved pronouns and registry IDs. Group every revision of one incident before splitting evaluation. Include a restricted note whose content may support an internal review but must not appear in a public handoff. A realistic holdout needs wrong-merge traps, not only easy repeated names.

Stage chains with reasons

Generate candidates within incident and environment scope, then retain the source span and match reason on each proposed edge. Mark edges stale when their source revision changes. Require review for high-impact ambiguous group or bridging decisions. The draft handoff may state that a referent is unresolved. Cross-document scope and access filtering run before any visible claim is assembled.

Gate the release

Score incorrect cross-incident merges, missing group members, identity-versus-bridging confusion, stale evidence and access leaks. Review final handoff sentences for claims about the wrong component or environment. Measure reviewer time and abstention, since forcing every pronoun into a chain can improve a pair metric while harming the handoff. Shadow the system on resolved incidents, then keep the previous chain build available for rollback.

Implementation

def visible_mentions(mentions, reader_roles, current_revisions):
    visible = []
    for mention in mentions:
        if mention["source_revision"] != current_revisions.get(mention["document_id"]):
            continue
        if not set(mention["roles_allowed"]) & set(reader_roles):
            continue
        visible.append(mention["mention_id"])
    return visible

mentions = [
    {"mention_id": "api-47", "document_id": "status", "source_revision": "r3",
     "roles_allowed": ["public", "responder"]},
    {"mention_id": "trace-82", "document_id": "private", "source_revision": "r4",
     "roles_allowed": ["responder"]},
]
assert visible_mentions(mentions, {"public"}, {"status": "r3", "private": "r4"}) == ["api-47"]

Performance and operating cost

Filtering m mentions is O(m) time and O(m) output space with small role sets. Candidate creation and reviewer checks add more work. Access should be enforced again at the final sentence-evidence boundary; a correct hidden chain can still leak if a generated handoff quotes its restricted source.

Common Mistakes

Showing a restricted antecedent in a public explanation.
Using a stale source revision for a new handoff.
Collapsing a group mention into one component.
Judging chain quality without inspecting final incident claims.