Project: release a runbook revision without stale answers

Last updated: 6 Oct 20265 min read

project

Classify a changed operational instruction, invalidate its derived evidence and verify public search and answers before release.

Set the revision task

An owner changes gateway-west rollback waiting time from 47 to 82 minutes and adds a production-only exception. The editorial system stages both source revisions and identifies changed claims. Search snippets, generated answers and saved handoffs that depended on the old passage must be marked stale. The system should not rewrite an unrelated staging instruction. Claim-level comparison distinguishes the threshold change from wording edits.

Build an impact fixture

Include old and new passages, a direct quote, a multi-passage answer, an index entry, an incident handoff and a restricted appendix. Add a grammar-only edit that should not cause the same rebuild. Reviewers annotate claim changes, environment scope and direct dependencies. Freeze the fixture before adjusting the diff rules. A full-document edit distance baseline can be compared, but cannot determine which outputs are safe to keep.

Stage withdrawal and rebuild

Map changed source passage IDs to downstream artifacts, withdraw stale public outputs, then rebuild from the approved revision. Keep prior artifact versions for audit. Re-run answer verification against active source spans, environment and access policy. Dependency traversal identifies affected descendants; claim verification checks that new wording does not exceed the source.

Gate the visible release

Test the public route, search snippet, API answer and saved handoff for the old threshold. Report missed stale outputs, false invalidations, time to withdraw and reviewer minutes. Release only when every affected artifact has an approved new version or a visible unresolved state. Roll back the source and derived artifacts together if the new instruction fails review; changing only the page while leaving answers cached is incomplete.

Implementation

def revision_release_state(affected_ids, artifact_states):
    pending = sorted(artifact_id for artifact_id in affected_ids
                     if artifact_states.get(artifact_id) not in {"approved", "withdrawn"})
    if pending:
        return {"state": "hold", "pending": pending}
    return {"state": "ready", "checked": len(affected_ids)}

affected = {"index-47", "answer-82", "handoff-91"}
states = {"index-47": "approved", "answer-82": "withdrawn",
          "handoff-91": "approved"}
assert revision_release_state(affected, states) == {"state": "ready", "checked": 3}
assert revision_release_state(affected, {**states, "handoff-91": "stale"})["state"] == "hold"

Performance and operating cost

Checking a set of affected artifacts is O(a log a) in the worst case because pending IDs are sorted, with O(a) temporary space. Source comparison, dependency traversal and answer verification add their own costs. Keeping a stale public answer hidden while rebuilding is safer than serving the old instruction for a few extra seconds.

Common Mistakes

Publishing the new page before withdrawing old generated answers.
Applying a production-only exception to staging results.
Treating a grammar edit as equal to a threshold change.
Rolling back the source document without rolling back its derived artifacts.