Few-shot prompting: select examples that cover decisions, not just easy cases

Decision in practice

A support team classifies refund tickets into approved, denied, or needs-review. Three examples cover a valid 31-day warranty claim, an expired warranty with no exception, and a case where the purchase date is absent. A fourth shows conflicting order identifiers. Each example names the policy field that drove its label. The candidate ticket is placed after the examples with a clear boundary. The team then tests 52 held-out tickets across the same classes, including rare conflicts, instead of scoring the four prompt examples as if they were independent proof of quality.

Example A: purchase age 31 days; policy window 44 days -> approved.
Example B: purchase age 61 days; no exception -> denied.
Example C: purchase date missing -> needs-review.
Candidate: classify ticket RT-582 and name the deciding field.

Performance and operating cost

Each extra example adds input tokens, latency, and sometimes distraction. The total context cost grows with example count and example length, while coverage can plateau or regress. Compare a zero-example baseline with a small, diverse set on the same held-out cases. If the task changes, refresh examples and evaluation together but preserve a stable core set for comparison. An example is a teaching artifact, not a policy source: the current policy still controls the answer.

Common Mistakes

• Do not choose only favorable examples.
• Do not leak the evaluation answer into the prompt examples.
• Do not let an old example override the current policy.

Connected lessons

• Prompt Engineering
• Prompt evaluation: test failures before rewriting the wording
• Prompt inputs: normalize records before asking for conclusions
• Output contracts: parse a result and preserve an explicit unknown state
• Project: evaluate an incident handoff prompt
• Prompt design decisions

Continue with: Few-shot examples: teach the boundary with near misses.