Project: build a reviewed runbook keyphrase index

Last updated: 6 Oct 20265 min read

project

Extract source-grounded phrases from runbooks, rank them for agent search and remove them when the source changes.

Set the product boundary

An incident runbook contains symptoms, commands and recovery steps. The index should expose phrases that help an authorized agent find the right section, not invent a diagnosis. Store phrase text, original span, source revision, access scope and extractor version. A phrase has no independent truth once its source is edited. Keep a link to the exact runbook section so an agent can inspect the surrounding instruction.

Create the review set

Write realistic search questions from agent tasks. Have reviewers mark relevant runbook sections and useful phrase candidates independently. Include nested terms, repeated titles, rare commands, renamed products and private ticket identifiers. Group documents by incident family before splitting. Candidate generation must keep original offsets; ranking must measure usefulness rather than frequency alone.

Build a minimal index

Extract bounded candidates and reject boilerplate and sensitive spans. Rank the remaining phrases, allow a small number per section and keep the source text for verification. Compare against title-only and sparse lexical baselines at a fixed review depth. Filter by tenant and access scope before displaying phrases or linked sections. Treat a missing source revision as an invalid index record.

Release and maintain

Shadow-build a new index when extraction rules or corpus revisions change. Check source counts, stale spans, retrieval gain, leakage and deletion propagation. Move the index alias only after the fixed audit passes, and retain the old safe version briefly for rollback. Monitor phrase click-through alongside false matches; clicks alone can reward misleading but attractive terms.

Implementation

def verified_phrase_record(document, span, source_revision):
    start, end = span
    if not source_revision or not 0 <= start < end <= len(document):
        raise ValueError("phrase source is not verifiable")
    phrase = document[start:end]
    return {"text": phrase, "start": start, "end": end,
            "source_revision": source_revision}

record = verified_phrase_record("Inspect queue lag.", (8, 17), "rev-47")
assert record["text"] == "queue lag"

Performance and operating cost

Span verification is O(1) plus O(length of phrase) to copy the text. Extraction across d documents costs at least the total token count and index writes; shadow builds temporarily double storage. The harder cost is reviewer time for query relevance and sensitive-term checks. Keep a fixed phrase budget per section and report retrieval gain per added index entry.

Common Mistakes

Publishing a phrase without a source revision or offset.
Indexing private ticket IDs from a copied runbook example.
Evaluating extraction against the same documents used to tune ranking.
Failing to remove phrases when a runbook section is deleted.