Retrieval prompts: rerank only authorized, current evidence

Last updated: 5 Oct 202611 min read

tutorial

Reranking reorders a candidate set; it does not grant access, repair missing documents, or decide that an obsolete procedure is current. Feed a reranker only passages already authorized for the caller. Before answer generation, compare each candidate's site, equipment identifier, effective version, and supersession status with the request. An older passage may remain useful for explaining a historical change but should not be presented as the active procedure. The prompt should ask the model to name a version conflict or insufficient evidence rather than blend steps from different manuals. A high relevance score is a ranking signal, not proof that a passage supports a final claim.

Operational case

Meridian's first-pass search returns three authorized P-47 passages. Manual revision R-8 is active; R-6 is archived and includes a reset torque that changed. A reranker favors R-6 because its wording mirrors the question. The version gate excludes R-6 from the active procedure while retaining its ID for the private audit trace. R-8 lacks one requested postcheck, so the assistant asks for the current postcheck source rather than borrowing it from the retired manual. The restriction on contract page C-91 remains unchanged throughout; its tempting similarity never makes it a reranker input.

Candidates: M-12/P47/R8 active; M-12/P47/R6 superseded.
Reranker scores relevance only among permitted candidates.
Version gate removes R6 from active instructions.
Missing postcheck in R8: mark unknown; request current evidence.
Never treat a score as authorization or factual support.

Performance and operating cost

Scoring K query-passage pairs adds O(K) reranker calls or one batched pass, with cost and latency proportional to candidate text. Version checks are O(K) metadata work. Limit K, but test that the answer-bearing authorized passage survives candidate generation before trimming. An extra model pass may improve ordering while raising expense; compare against a no-rerank baseline on held-out questions. When entitlement changes between retrieval and generation, recheck access before context assembly or discard the stale packet.

Common Mistakes

Do not rerank restricted passages and filter only after generation.
Do not promote a superseded manual because its wording scores highly.
Do not call reranker relevance a claim-support check.

Connected lessons

prompt engineering

authorized retrieval