Skip to content
AITroveRead. Build. Understand.
Reading list

Production prompt engineering

Release tests, adversarial inputs, cost, privacy, and language parity.

Release tests, adversarial inputs, cost, privacy, and language parity. Start with the decision you need to make, then use the linked material to build a checkable result.

Learning path

Connect the path

The foundations define the task and trust boundary. Later modules test whether a result should be used, revised, or sent to a person.

Experiments, releases, and review controls

These lessons turn prompt wording into a tested workflow. Apply each rule to a real input boundary, then inspect whether the decision and any side effect still match the task contract.

Decision gates and downstream validation

A reliable answer needs the right information, a valid calculation or decision, and a destination that treats generated content as data.

Executable evaluation labs

Use these short programs to check evidence references, abstention, judge consistency, and critical release cases. A passing structural check still needs semantic review where the claim depends on a passage.

Tool result and effect controls

Contain instructions in tool data and reconcile ambiguous effects.

Negative controls

Test cases that should not produce an answer or external effect.

Delivery and telemetry

Connect the tested prompt to a merge gate, a complete release bundle, and a trace that preserves decisions without copying private payloads.

Evaluation governance

Test paired outcomes, labels, attack boundaries, live exposure, and case revisions.

Safety decision boundaries

Route requests, refuse narrowly, validate output, and reserve high-impact decisions for review.

Release review packet

Bring case-level checks, costs, promotion, and rollback into one decision.

Cross-locale evaluation

Pair semantic cases across languages and compare decisions.

Accessible output evaluation

Check artifact failures and rendered user tasks before promotion.

Browser-agent recovery

Resolve an uncertain submit with a receipt or authoritative state.

Voice-agent evaluation

Test event timing, noise, receipts, and final account state.

Synthetic evaluation-case engineering

Generate cases from a contract, validate labels, protect privacy, and keep holdouts independent.

Generated media release review

Check continuity, captions, transcript, and the actual published asset.

Spreadsheet calculation release gate

Recalculate the final workbook and compare it with the reviewed artifact.

Specialist run release gate

Bound timeouts and retries, review the trace, and name missing results.

Research brief release gate

Tie the final claim to the frozen packet, exclusions, calculations, and reviewer.

Tutor state and release checks

Minimize learner data and test new-case transfer before release.

UI release evidence

Compare rendered states and test keyboard, semantics, and truthful feedback.

PDF export release

Verify the delivered file's text, pages, structure, and approved claims.

Meeting recap release

Check access, sensitive material, correction routes, and side-effect receipts.

Architecture release evidence

Load-test the chosen contract and stage its rollout with rollback ownership.

API docs release evidence

Check links, examples, shared status lists, and deployed behavior.

Infrastructure release evidence

Verify live resources, behavior, drift, and rollback limits after apply.

Incident handoff and closure

Keep an action ledger, verify recovery, and correct earlier claims.

Retrieval release gates

Test revocation, deletion, timeouts, quality, and denied-content leakage.

Memory deletion and release

Confirm every usable copy is gone and test persistent attack boundaries.

Recurring task controls

Deduplicate retries, bound alerts and effects, and retain run evidence.

Terminal-agent release controls

Interpret exit status, preserve changes, respect permissions, and verify final claims.

Chart release checks

Store reproducible specs and review rendered and nonvisual outputs.

Email and calendar effect gates

Review recipients, receipts, event updates, and recurrence scope.

Route feasibility and release

Compute windows, inspect map points, and gate dispatch effects.

Qualitative analysis and decisions

Adjudicate labels, retain counterexamples, reconcile people, and gate claims.

Forecast evaluation and decisions

Test baselines, distinguish scenarios, check intervals, and gate stock actions.

Security triage decisions

Apply severity rules, quarantine log instructions, and gate containment.

Survey testing and release

Pilot the form, reconcile counts, protect open text, and bound claims.

Support decisions and handoff

Verify permissions, ask focused questions, and separate draft from effect.

Accessibility review review

Verify state, arithmetic, and release boundaries.

Product analytics review

Verify state, arithmetic, and release boundaries.

Localization QA review

Verify state, arithmetic, and release boundaries.

Product experiments review

Verify state, arithmetic, and release boundaries.

Invoice exception review review

Verify comparisons and release boundaries.

Database backfill review

Verify comparisons and release boundaries.

Search relevance evaluation review

Verify comparisons and release boundaries.

Annotation quality workflow review

Verify comparisons and release boundaries.

Curriculum

Release tests, adversarial inputs, cost, privacy, and language parity.

  1. 1Prompt injection: test untrusted content at every boundary
  2. 2Evaluation sets: measure the failure cases that matter
  3. 3Model judges: calibrate rubrics and swap candidate order
  4. 4Prompt releases: version the whole decision path and keep a rollback
  5. 5Prompt budgets: trade output quality against cost and tail latency
  6. 6Prompt privacy: send only the fields needed for the task
  7. 7Multilingual prompts: test policy meaning across languages

Test these decisions with representative input, explicit failure states, and recorded versions.

  1. 1Prompt caching: reuse stable context with a versioned expiry
  2. 2Metamorphic tests: verify behavior when harmless details change
  3. 3Human handoff: preserve evidence and the reason for uncertainty
  4. 4Prompt traces: connect an answer to its inputs, checks, and effects
  5. 5Model migration: replay contracts before changing providers or versions
  6. 6Fairness checks: test equivalent cases across groups and wording

Locate the failing layer, then test the repaired decision boundary.

  1. 1Selective answers: measure when to abstain
  2. 2Evaluation leakage: keep the release test independent
  3. 3Generated output: validate again at the destination boundary

Run small deterministic checks before interpreting model output or promoting a prompt.

  1. 1Code lab: reject unknown evidence IDs
  2. 2Code lab: score answered cases and abstentions
  3. 3Code lab: find pairwise judge order changes
  4. 4Code lab: block a critical prompt regression

Contain instructions in tool data and reconcile ambiguous effects.

  1. 1Tool results: keep returned text in the data lane
  2. 2Tool effects: reconcile receipts before retrying

Test cases that should not produce an answer or external effect.

  1. 1Negative controls: test the answer that should not be produced

Carry prompt evaluation into CI, promotion, and privacy-aware operations.

  1. 1Prompt changes in CI: test the merge candidate
  2. 2Prompt release artifacts: version the whole decision path
  3. 3Prompt telemetry: measure failures without copying private payloads

Test paired outcomes, labels, attack boundaries, live exposure, and case revisions.

  1. 1Paired prompt evaluation: count changes, then inspect uncertainty
  2. 2Evaluation labels: adjudicate disagreement before scoring a release
  3. 3Adversarial case mutations: test the boundary, not a magic phrase
  4. 4Online prompt experiments: define exposure and stop rules first
  5. 5Evaluation case ledgers: revise labels without erasing history

Route requests, refuse narrowly, validate output, and reserve high-impact decisions for review.

  1. 1Request risk routing: classify the action before choosing a response
  2. 2Refusal contracts: decline the unsafe effect and preserve useful help
  3. 3Sensitive output gates: check the rendered answer before release
  4. 4System prompts: treat instructions as guidance, not a secret vault
  5. 5High-impact advice: route uncertain decisions to qualified review

Bring case-level checks, costs, promotion, and rollback into one decision.

  1. 1Prompt release review: assemble the decision packet

Pair semantic cases across languages and compare decisions.

  1. 1Cross-locale evaluation: compare decisions, not word-for-word text

Check artifact failures and rendered user tasks before promotion.

  1. 1Accessible output evaluations: measure failures by artifact and user task

Resolve an uncertain submit with a receipt or authoritative state.

  1. 1Browser effect recovery: resolve ambiguous submissions before retrying

Test event timing, noise, receipts, and final account state.

  1. 1Real-time voice evaluations: test timing and state, not transcript fluency

Generate cases from a contract, validate labels, protect privacy, and keep holdouts independent.

  1. 1Synthetic evaluation cases: mutate a contract, not a customer’s record
  2. 2Synthetic case oracles: reject invalid labels before scoring a model
  3. 3Synthetic case coverage: count distinct decisions, not rewritten sentences
  4. 4Synthetic cases are not automatically private
  5. 5Synthetic evaluation holdouts: stop the generator from teaching the test

Check continuity, captions, transcript, and the actual published asset.

  1. 1Generated video release: inspect continuity, claims, and alternatives

Recalculate the final workbook and compare it with the reviewed artifact.

  1. 1Spreadsheet release checks: recalculate, reconcile, then inspect

Bound timeouts and retries, review the trace, and name missing results.

  1. 1Specialist workflows: bound retries and review the full trace

Tie the final claim to the frozen packet, exclusions, calculations, and reviewer.

  1. 1Research synthesis release: expose the method and its limits

Minimize learner data and test new-case transfer before release.

  1. 1Tutor release checks: minimize learner state and test transfer

Compare rendered states and test keyboard, semantics, and truthful feedback.

  1. 1UI release prompts: compare appearance and verify access

Verify the delivered file's text, pages, structure, and approved claims.

  1. 1PDF export prompts: verify the delivered file again

Check access, sensitive material, correction routes, and side-effect receipts.

  1. 1Meeting recap release: review scope, access, and corrections

Load-test the chosen contract and stage its rollout with rollback ownership.

  1. 1Architecture release: require evidence, rollback, and ownership

Check links, examples, shared status lists, and deployed behavior.

  1. 1API docs release: test links, examples, and live version

Verify live resources, behavior, drift, and rollback limits after apply.

  1. 1Infrastructure release: verify live state and rollback limits

Keep an action ledger, verify recovery, and correct earlier claims.

  1. 1Incident handoffs: preserve actions and decisions
  2. 2Incident resolution: verify recovery and correct the record

Test revocation, deletion, timeouts, quality, and denied-content leakage.

  1. 1Retrieval release: test permission changes and deletion

Confirm every usable copy is gone and test persistent attack boundaries.

  1. 1Memory prompts: expire and delete every usable copy
  2. 2Memory release: test recall, poisoning, and isolation

Deduplicate retries, bound alerts and effects, and retain run evidence.

  1. 1Recurring prompts: survive retries without duplicate effects
  2. 2Recurring prompts: notify only on actionable changes
  3. 3Recurring prompts: renew authority for sensitive actions
  4. 4Recurring prompts: make every run replayable and auditable

Interpret exit status, preserve changes, respect permissions, and verify final claims.

  1. 1Terminal agents: distinguish exit status from useful output
  2. 2Terminal agents: request only the missing execution scope
  3. 3Terminal agents: keep the patch within owned files
  4. 4Terminal agents: make the final claim match verified checks

Store reproducible specs and review rendered and nonvisual outputs.

  1. 1Chart prompts: produce a reproducible spec and data lineage
  2. 2Chart prompts: review rendered and nonvisual outputs

Review recipients, receipts, event updates, and recurrence scope.

  1. 1Email prompts: review To, Cc, Bcc, and reply scope
  2. 2Email prompts: separate drafted commitments from sent messages
  3. 3Calendar prompts: distinguish create, update, and cancellation
  4. 4Calendar prompts: scope recurrence changes to the right instances

Compute windows, inspect map points, and gate dispatch effects.

  1. 1Location prompts: compute stop-window feasibility before recommending
  2. 2Location prompts: review the map and gate dispatch effects

Adjudicate labels, retain counterexamples, reconcile people, and gate claims.

  1. 1Interview prompts: keep coder disagreement visible
  2. 2Interview prompts: build a theme evidence ledger
  3. 3Interview prompts: count people, not repeated excerpts
  4. 4Interview prompts: turn bounded findings into testable decisions

Test baselines, distinguish scenarios, check intervals, and gate stock actions.

  1. 1Forecast prompts: compare against a rolling baseline
  2. 2Forecast prompts: keep scenarios separate from predictions
  3. 3Forecast prompts: label intervals and check coverage
  4. 4Forecast prompts: gate inventory recommendations on evidence

Apply severity rules, quarantine log instructions, and gate containment.

  1. 1Security alert prompts: apply scoped severity and escalation rules
  2. 2Security alert prompts: quarantine instructions embedded in logs
  3. 3Security alert prompts: gate containment and verify receipts

Pilot the form, reconcile counts, protect open text, and bound claims.

  1. 1Survey prompts: pilot the instrument before fielding
  2. 2Survey prompts: reconcile invitations, submissions, and item denominators
  3. 3Survey prompts: code open text without exposing respondents
  4. 4Survey prompts: release findings with explicit scope

Verify permissions, ask focused questions, and separate draft from effect.

  1. 1Support prompts: verify entitlement and requester authority
  2. 2Support prompts: ask only the question that changes the next step
  3. 3Support prompts: separate reply, handoff, and resolution

Verify state, arithmetic, and release boundaries.

  1. 1Accessibility prompts: connect form errors and status changes
  2. 2Accessibility prompts: measure contrast and describe image purpose
  3. 3Accessibility prompts: inspect zoom and reflow states
  4. 4Accessibility prompts: write reproducible findings and retest fixes

Verify state, arithmetic, and release boundaries.

  1. 1Analytics prompts: pin cohort windows and time zones
  2. 2Analytics prompts: review read-only queries and totals
  3. 3Analytics prompts: explain segment gaps without inventing causes
  4. 4Analytics prompts: turn a metric packet into a bounded decision

Verify state, arithmetic, and release boundaries.

  1. 1Localization prompts: render typed dates, numbers, and units
  2. 2Localization prompts: reconcile terminology and tone
  3. 3Localization prompts: use pseudolocale and real layout checks
  4. 4Localization prompts: gate release on catalog and rendered retests

Verify state, arithmetic, and release boundaries.

  1. 1Experiment prompts: reconcile missing outcomes by assigned unit
  2. 2Experiment prompts: separate effect size from uncertainty
  3. 3Experiment prompts: investigate slices and guardrail movement
  4. 4Experiment prompts: gate rollout on a reproducible packet

Verify comparisons and release boundaries.

  1. 1Invoice prompts: match order, receipt, and bill
  2. 2Invoice prompts: distinguish duplicate ingests from credits
  3. 3Invoice prompts: route exceptions before approval

Verify comparisons and release boundaries.

  1. 1Backfill prompts: account for writes during migration
  2. 2Backfill prompts: compare count and field-level parity
  3. 3Backfill prompts: hold cutover until fallback is rehearsed

Verify comparisons and release boundaries.

  1. 1Search prompts: calculate ranking metrics on judged pairs
  2. 2Search prompts: interpret clicks and gate a ranking release

Verify comparisons and release boundaries.

  1. 1Annotation prompts: adjudicate conflicts and version the rule
  2. 2Annotation prompts: protect holdouts and sample drift
Storage details