Retrieved evidence: reconcile versions and conflicting facts

Last updated: 7 Oct 20269 min read

tutorial

Retrieval quality includes freshness, authority, and sufficiency. Attach source ID, version, effective date, and access scope to each passage. When two passages disagree, establish which policy period covers the case; do not choose the one that appeared first or reads more clearly. If the conflict cannot be resolved from metadata, return an uncertainty state with both IDs. The prompt should ask for a decision only after the application has filtered inaccessible and expired material.

Decision in practice

A freight-credit assistant finds policy CP-18 from a retired folder and policy CP-23 effective for current shipments. The customer request concerns a shipment dated after CP-23 became effective, so only CP-23 governs. A second case lacks the shipment date and cannot choose between them. The assistant reports that gap rather than using the top search hit. A reviewer checks that the cited section ID and effective window match the case, then either confirms the credit or requests the missing date.

Candidate CP-18: retired, effective through 2025-12-31.
Candidate CP-23: effective from 2026-01-01.
Case: shipment date 2026-02-14 -> CP-23.
Case without date -> review; cite both conflicting versions.

Performance and operating cost

Metadata filtering can reduce tokens sent to the model but needs a reliable document lifecycle. For M retrieved passages, checking their metadata is O(M) before ranking or synthesis. Stale indexes can still return retired text, so evaluate delayed index updates and conflicting versions. Keep the original document and extracted passage tied to the same version. A cache must expire or invalidate when the governing policy changes, even if the text has a high reuse rate.