Project: verify incident claims against reviewed source revisions

Last updated: 6 Oct 20265 min read

project

Build a claim-review service that checks supporting spans, flags temporal conflict and never turns a high similarity score into an approved fact.

Define the question

At shift handoff an agent writes “The callback retry completed at 14:47.” The service must say whether current authorized incident evidence supports that exact claim, contradicts it or leaves it unresolved. Return source revisions, offsets, observed times and review state. An unsupported claim is a draft requiring human action, not a fact with a lower confidence number.

Assemble the audit set

Use reviewed incident claims with negation, quantity changes, mistaken identities, explicit retractions and multiple source versions. Keep all notes from one incident together when splitting; reserve later incidents for final audit. Store the minimal evidence span for each non-neutral label. The premise-hypothesis contract defines labels; temporal review resolves multi-source status.

Measure the full route

Retrieve permitted passages first, then score premise-claim pairs and apply current-revision policy. Report evidence retrieval recall, per-label errors, false accepted claims, conflicts sent to review and access failures. Compare a deterministic status-rule baseline with a learned inference model. A classifier that scores well on gold passages but misses later corrections fails the operational release gate.

Connect the handoff

Only reviewed current-supported claims may enter an automatically prepared incident summary. Keep disputed claims in a restricted queue with their source windows and decision history. Invalidate an approved status when the underlying revision is retired. The handoff summary project consumes the accepted claim ledger, but still requires a human signoff on the final text.

Implementation

def publishable_claim(inference_label, source_current, source_authorized,
                      reviewer_approved, conflicting_current_sources):
    return (inference_label == "entails" and source_current
            and source_authorized and reviewer_approved
            and not conflicting_current_sources)

assert publishable_claim("entails", True, True, True, False)
assert not publishable_claim("entails", True, True, True, True)

Performance and operating cost

The final gate is O(1) time and space. Retrieval, inference and adjudication dominate cost; track each stage separately. Source updates can invalidate earlier approvals, so cache claim status only with source revision and review version. A larger model cannot compensate for missing evidence or stale access checks.

Common Mistakes

Publishing a claim because one retrieved passage uses similar words.
Ignoring a later correction in a separate note.
Treating current access as permanent for cached evidence.
Measuring model labels but not end-to-end false accepted claims.