Retrieval corpus contracts: identity, permissions and document lifecycle

Last updated: 5 Oct 20265 min read

tutorial

A retrieval system must know which document revision a passage came from and who may read it before indexing begins.

Treat documents as governed records

An internal support handbook may contain policy pages, drafts and retired procedures. Assign stable document ID, revision, effective time, owner and visibility scope. Do not merge a current policy with a superseded one merely because both mention the same product. The source contract] should preserve an immutable copy for replay and audit.

Carry permissions into the index

Every indexed passage inherits the document’s authorization policy. Retrieval must filter candidates for the current user before a passage can enter the model context or a cache. Filtering only after generation is too late. Test two employees with different scopes against the same question and confirm neither receives inaccessible passages, even in suggestions or logs.

Model deletion and replacement

A document removal should invalidate all derived chunks, embeddings and cached answers tied to that revision. An update should produce a new revision, then switch the visible index generation after the new representation is complete. Index publication] prevents readers from seeing a half-updated corpus.

Audit a complete path

For a retrieved passage, trace document ID, revision, chunk ID, permissions and index generation. Add fixtures for an expired policy, revoked access and an identical title in two departments. A keyword match is not evidence that the user may read the result.

Implementation

def visible_passages(passages, principal_scopes, query_time):
    permitted = []
    for passage in passages:
        if not passage.active_from <= query_time:
            continue
        if passage.active_until and query_time >= passage.active_until:
            continue
        if not passage.allowed_scopes.intersection(principal_scopes):
            continue
        permitted.append(passage)
    return permitted

Performance and operating cost

Filtering K candidate passages is O(K) in this example; indexed permission filters can reduce work earlier. Retaining revisions consumes storage but supports correct deletion, audit and rollback.

Common Mistakes

Do not filter permissions only after generation.
Do not leave retired policy chunks searchable.
Do not use a title as a stable document identity.