A test-generation prompt can propose cases from a requirement, interface contract, and failure history. The expected result must come from those sources, not from running the implementation under test and copying its output. Define boundary values, invalid inputs, duplicate operations, and state transitions where relevant. A passing generated test is weak if it merely restates the current code. Review at least one deliberately faulty variant or historical regression to see whether the test would fail for the right reason. Keep fixtures isolated so a test result does not depend on an earlier run.
Make this comfortable
Generated tests: verify the oracle before trusting coverage
Operational case
A wallet API promises that a second request with the same transfer key returns the first receipt without moving money again. The assistant proposes a test for transfer key TR-694, amount 43.75, and an account balance of 120.00. The first call should leave 76.25 and return receipt RC-694; the second call should return the same receipt and leave 76.25. A broken variant that debits twice would leave 32.50. If the generated assertion only checks that both calls return status 200, it misses the defect despite passing. The test must assert both receipt identity and final balance.
Initial balance: 120.00; transfer TR-694: 43.75.
First call: receipt RC-694; balance 76.25.
Same key again: receipt RC-694; balance still 76.25.
Fault injection: second debit -> 32.50; test must fail.
Oracle: transfer contract, not current endpoint output.Performance and operating cost
For N candidate cases and M fault variants, a full matrix can require O(NM) executions, so focus mutations on meaningful boundaries and past failures. Fixture setup and external calls may dominate runtime; isolate them before multiplying cases. A mutation score is useful only when the fault variants represent behavior the contract forbids. Record which case killed which fault, not just line coverage. The model can draft cases quickly, but a human or trusted specification must approve the oracle before it becomes a release gate.
Common Mistakes
- Do not use the current buggy implementation as the expected-result source.
- Do not count a status-only assertion as an idempotency test.
- Do not reuse mutable account state across tests without reset.
Connected lessons
- Prompt engineering applications
- Prompt Engineering
- Coding prompts: name the files, behavior, and proof
- Python mutation testing: prove a suite rejects selected wrong implementations
- CI test data: give each run an isolated ownership scope
- Diff-scoped code review: make every finding reproducible
- Incident triage prompts: build a timestamped evidence ledger
- Schema migration prompts: check old and new clients together
- Dependency upgrade prompts: trace behavior, not only versions
- Project: review a checkout release from diff to incident
- Prompt engineering for software delivery
Continue with: Tutor feedback: identify the specific reasoning step.
Continue with: Responsive UI prompts: test states, not one viewport.
Continue with: API examples: validate payloads and strip credentials.
Continue with: Terminal agents: distinguish exit status from useful output.
prompt engineering
software delivery