Long context: make inclusion and truncation testable

Last updated: 5 Oct 20268 min read

tutorial

Long prompts need an explicit budget for instructions, evidence, examples, and output. Truncation is a data-loss decision, not a harmless formatting change. Select passages by relevance, freshness, and required fields before packing them, then record what was omitted. Stable identifiers help test whether the answer used the right span. Position effects vary by model and task, so evaluate facts placed early, in the middle, and near the end instead of assuming one fixed ordering rule. A summarization chunk should retain the uncertainty and source IDs that later stages need.

Decision in practice

A procurement assistant compares a 190-page vendor agreement with a renewal notice. A naive prompt appends the full agreement, then the notice, and silently drops the middle clauses when a token limit is hit. The revised workflow extracts the termination, price-adjustment, and notice-period sections with page IDs, plus neighboring definitions. Its manifest records omitted sections and the total token count. A test set asks about terms located at different positions in the original document. If the relevant clause was not included, the assistant reports that the packet is insufficient instead of filling the gap from memory.

Budget: instructions 900 tokens; evidence 6,400; examples 700; output 900.
Required spans: termination, price adjustment, notice period, definitions.
Manifest: document version, page IDs, omitted sections, token count.
Failure: required span missing -> abstain or retrieve again.

Performance and operating cost

Token counting and passage selection add preprocessing cost, but uncontrolled overfill can waste the entire model call. For N candidate passages, a simple relevance sort is O(N log N); more elaborate reranking adds model or search expense. Measure answer accuracy by passage position and by omitted-field class. Summaries can compress context, but a lossy summary may remove the exact qualifier that decides a contract question. Reserve output capacity so the model is not cut off mid-answer.

Common Mistakes

Do not assume a long window guarantees complete recall.
Do not let truncation silently discard a required clause.
Do not summarize away version, scope, or uncertainty markers.

Connected lessons

Continue with: Retrieval prompts: pack evidence and map claims.

prompt engineering

tutorial