Retrieval release: test permission changes and deletion

Last updated: 4 Oct 202611 min read

tutorial

A secure retrieval release needs positive and negative tests. Verify that authorized users can find the right active passage and that unauthorized users cannot receive its text, title, summary, or citation. Test a grant change after a cached query, a document deletion, a superseded revision, and a retriever timeout. Deletion must cover the source, index, caches, and saved context packets according to the system's retention contract; removing one search hit is not enough. Evaluate retrieval recall, final claim support, latency, and denied-content leakage separately. A good answer score cannot compensate for a permission failure, and a secure refusal does not prove the authorized path works.

Operational case

Meridian removes a contractor's access to site M-12 while a previous retrieval packet is cached. The release test repeats the same P-47 question under the contractor's verified identity and requires no M-12 text or document title in the response. A technician with continuing access must still retrieve active R-8 sections 4.1 and 4.2. The team then deletes a retired R-6 file and checks the index, result cache, and context-packet store for its ID. A timed-out entitlement lookup must fail closed. Only after these cases pass does the release owner approve the prompt-and-retriever bundle.

Allowed case: technician sees active R8/4.1 and R8/4.2.
Denied case: contractor sees no M-12 passage, title, or citation.
Revocation case: cached packet cannot bypass new permission state.
Deletion case: R6 absent from source, index, and defined caches.
Timeout case: fail closed; record a private operational alert.

Performance and operating cost

A release suite with U user roles and Q queries can require O(UQ) retrieval checks; target high-risk role and document pairs rather than only average relevance. Deletion checks add storage-specific scans and may need an asynchronous completion window. Record the permission version and index version used by each test so a pass is tied to an actual state. A zero-leak assertion belongs beside quality metrics as a hard gate, because averaging it into a helpfulness score could hide one severe disclosure.

Common Mistakes

Do not treat cache invalidation as optional after access revocation.
Do not measure only authorized-answer quality.
Do not claim deletion is complete while old packets remain available.

Connected lessons

Continue with: Search prompts: diagnose candidate coverage and zero results.

prompt engineering

authorized retrieval