Project: index runbook sections with revision-safe passages

Last updated: 6 Oct 20265 min read

project

Build a section-aware runbook index that returns the exact supporting passage and invalidates it when the source changes.

Define the retrieval product

An operator searches versioned runbooks for a recovery condition. The result should show the relevant section, heading context and current document revision, not just a disconnected sentence. A retrieved passage is evidence for reading, not permission to execute a command. Preserve access scope and original offsets so the operator can open the entire surrounding section.

Create difficult queries

Include questions whose answer spans a warning and the next step, a table with repeated headings, an old runbook containing a conflicting limit and an exact incident code. Reviewers identify the minimum supporting section and whether a candidate passage is sufficient on its own. Group document families before splitting. Boundary design supplies the structure; revision auditing supplies stable release checks.

Build and compare

Create a heading-aware index and a fixed-window baseline from the same documents. Compare supporting-passage recall, wrong-section hits, duplicate results, latency and storage. Reconstruct the section around a selected result after enforcing source access. Keep table headers and caution notes attached. A passage that names the right command but omits its prohibition should count as an unsafe retrieval result.

Release and maintain

Build a shadow index when a runbook or chunker changes. Replay reviewed questions and cached answer citations against the new source revisions. Reject a promotion with orphaned citations or deleted-source hits. Move the alias only after access, deletion and retrieval checks pass. Keep a reversible prior version that excludes documents removed under retention policy.

Implementation

def current_passage(passage, source_revisions, allowed_documents):
    document_id = passage["document_id"]
    return (document_id in allowed_documents
            and source_revisions.get(document_id) == passage["revision"]
            and bool(passage["section_path"]))

candidate = {"document_id": "rb-47", "revision": "r8",
             "section_path": ("Recovery", "Retry policy")}
assert current_passage(candidate, {"rb-47": "r8"}, {"rb-47"})
assert not current_passage(candidate, {"rb-47": "r9"}, {"rb-47"})

Performance and operating cost

The current-revision and access check is O(1) average time per candidate; checking k retrieved passages is O(k). Building both index versions temporarily raises storage and indexing cost. The decisive metric is whether agents reach the correct, current supporting section without seeing retired or unauthorized material.

Common Mistakes

Showing an old passage because its text still matches the query.
Retrieving a command without its qualifying warning.
Treating a passage-level ACL copy as the current source policy.
Promoting a new chunker before cached citations are replayed.