Project: segment incident notes without losing evidence spans

Last updated: 7 Oct 20265 min read

project

Build a sentence-boundary release gate for incident notes, with abbreviation review, exact offsets and downstream claim checks.

Create the fixture

Use fictional on-call notes with service IDs, decimal latencies, release versions, abbreviations and pasted log fragments. Annotate sentence start and end positions in the immutable source text. Include a note where “svc.” precedes a service name and another where an abbreviation ends a sentence. Keep the same incident’s revisions in one data split. Boundary candidates establish the baseline policy.

Segment without rewriting

Protect known code and URL spans, propose punctuation boundaries, score ambiguous candidates and return source offsets. Preserve original whitespace and punctuation; show unresolved boundaries to reviewers with nearby context. Run a mechanical coverage check before any NLP downstream task sees the spans. Offset regression detects gaps after normalization or parser changes.

Test the later decisions

For each note, compare service entities, negated claims and action conditions with gold annotations before and after segmentation. A split that isolates “do not” from the action can be more damaging than two harmless merged sentences. Report boundary precision and recall alongside downstream claim accuracy. Hold a release if errors concentrate in high-risk runbooks even when the average score rises.

Release with a correction path

Keep the abbreviation registry version, source revision, model version and reviewer edits. If a reviewer changes a boundary, invalidate derived passage indexes and extracted claims that depended on the old span. The code below checks whether a note can enter downstream processing; it does not claim to decide boundaries. Add human review when ambiguous candidates affect actions or negation.

Implementation

def admit_segmented_note(note, spans, unresolved_positions):
    if unresolved_positions:
        return {"state": "review", "reason": "uncertain-boundary"}
    cursor = 0
    for start, end in spans:
        if not 0 <= start < end <= len(note) or start < cursor:
            return {"state": "review", "reason": "invalid-offsets"}
        if note[cursor:start].strip():
            return {"state": "review", "reason": "lost-text"}
        cursor = end
    if note[cursor:].strip():
        return {"state": "review", "reason": "lost-text"}
    return {"state": "ready-for-claim-check", "segments": len(spans)}

note = "Cache recovered. Check queue."
assert admit_segmented_note(note, [(0, 16), (17, len(note))], [])["state"] ==     "ready-for-claim-check"
assert admit_segmented_note(note, [(0, 16)], [])["reason"] == "lost-text"

Performance and operating cost

The gate takes O(n+s) time across n source characters and s ordered spans, with O(1) extra space. A claim checker has additional model and review cost. A ready state only says the note is structurally intact; it does not certify that every chosen boundary is linguistically correct.

Common Mistakes

Letting a gap of non-whitespace text pass unnoticed.
Treating a clean displayed sentence as proof its offset is correct.
Reusing boundaries after the source note is edited.
Ignoring how a split changes negation or action extraction.