Streaming responses: validate the complete artifact before an effect

Last updated: 7 Oct 20269 min read

tutorial

Streaming can show useful text early, but a parser should wait for a complete response before accepting structured data or triggering an action. Mark partial content as provisional in the interface. Accumulate bytes within a bounded buffer, handle disconnects, and validate the final object against schema and business rules. A visible word such as approved is not a completed authorization. For long generated reports, users may read partial prose while the application keeps download, publication, and financial effects disabled until the final check passes.

Decision in practice

An insurance assistant streams a claim review. The first chunk says 'approval likely'; the last chunk cites a missing receipt and changes the decision to review. The UI labels the stream as drafting and shows no approve button tied to the partial text. The server waits for the terminal event, validates the final JSON decision and evidence IDs, then presents the review state. A simulated disconnect after the first chunk creates an incomplete artifact with no effect. The case remains queued for retry or human review.

Stream state: receiving -> complete -> validated -> displayable decision.
Disconnect before complete: incomplete; no effect.
Final object: review, evidence_ids=[note-18], missing_receipt=true.
Action gate: only validated final state can reach an executor.

Performance and operating cost

Streaming can improve perceived responsiveness without reducing total generation cost. Buffering requires memory proportional to the bounded response size; set a cap before allocation grows without limit. A second validation pass adds small latency after the final chunk but prevents partial-object actions. Measure disconnect rate, parse failures, and the gap between first token and validated completion. Keep a request ID so retries can avoid duplicating an effect already committed elsewhere.

Common Mistakes

Do not parse an unfinished JSON prefix as a decision.
Do not allow an early sentence to authorize a write.
Do not show a disconnected draft as a completed answer.

Connected lessons

prompt engineering

tutorial