Project: build a reviewable argument map for an incident change

Last updated: 6 Oct 20265 min read

project

Extract a proposed operational change, supporting and opposing reasons, target-specific stances and unresolved evidence before writing a release summary.

Set the review task

A proposal raises gateway-west rollback waiting time to 82 minutes. One operator cites incident counts; another accepts the latency result but disputes the rollback procedure. The review output must not flatten those positions into one positive or negative label. Preserve the proposal revision, speaker turns and exact source spans. Argument units separate the proposed action from reasons offered for and against it.

Annotate the fixture

Include a claim, a supporting metric, an objection to a subclaim, a quoted old policy and a later correction. Reviewers tag unit boundaries, relation direction, speaker, target, time and evidence-verification status. Add an ambiguous “needs more testing” note that should remain unresolved. Freeze the annotation policy and adjudicate disagreements before scoring; otherwise the system can appear wrong simply because target scope shifted between reviewers.

Construct the map

Store nodes as immutable document-revision spans and edges as directed support or attack relations. Keep target-specific stance records separate from evidence truth checks. A cited incident count is not confirmed merely because it supports a claim in the writer’s argument. Check the metric against the underlying incident ledger before a release summary uses it. Stance records preserve conditional support and explicit opposition.

Gate the summary

A release summary can include a claim only when its source revision is current, its support relation is reviewable and any numeric evidence it relies on has been verified. Otherwise mark it unresolved and show the reason to the reviewer. Report missed objections, false support edges, attribution mistakes and unsupported numeric claims. The code below checks a tiny final gate; it does not infer argument structure from raw prose. Summary verification checks the wording after the map is approved.

Implementation

def releasable_claims(claims, current_revision):
    accepted = []
    held = []
    for claim in claims:
        reasons = []
        if claim["revision_id"] != current_revision:
            reasons.append("stale-revision")
        if not claim["relation_reviewed"]:
            reasons.append("unreviewed-relation")
        if claim["uses_numeric_evidence"] and not claim["evidence_verified"]:
            reasons.append("unverified-number")
        (held if reasons else accepted).append(
            {"claim_id": claim["claim_id"], "reasons": reasons})
    return {"accepted": accepted, "held": held}

claims = [
    {"claim_id": "change-47", "revision_id": "r8",
     "relation_reviewed": True, "uses_numeric_evidence": True,
     "evidence_verified": False},
    {"claim_id": "change-82", "revision_id": "r8",
     "relation_reviewed": True, "uses_numeric_evidence": False,
     "evidence_verified": False},
]
result = releasable_claims(claims, "r8")
assert result["held"] == [{"claim_id": "change-47",
                            "reasons": ["unverified-number"]}]
assert result["accepted"] == [{"claim_id": "change-82", "reasons": []}]

Performance and operating cost

The final gate scans c claims in O(c) time and uses O(c) space for its returned decisions. Span extraction, relation labeling, evidence checks and adjudication add separate work. A passing gate only says the recorded fields meet the policy; the team must still review whether those fields faithfully describe the source text.

Common Mistakes

Equating a cited metric with verified evidence.
Summarizing a quoted objection as the proposal owner’s position.
Treating support for a latency result as support for the rollback plan.
Releasing a claim from an obsolete proposal revision.