Calling a record synthetic does not establish a privacy guarantee. A model asked to imitate real tickets may retain names, tracking numbers, addresses, or rare combinations that identify a person. Prefer building fictional cases from an abstract rule schema. If examples from production are necessary, minimize and transform them under an approved data process before generation, then screen outputs for exact identifiers and close matches. Review rare narratives as well as obvious fields; a distinctive incident can be identifying even without a name. Keep the source of each case and the transformation decision in a restricted ledger. A privacy-preserving statistical dataset is a separate engineering problem; prompt-based fixture generation alone does not provide formal protection.
Synthetic cases are not automatically private
Operational case
A reviewer notices that generated ticket RT-648 includes a real parcel reference and an unusual delivery note copied from a support transcript. The generator changed the customer's name, but the reference and story still point to one person. The team removes that case from the shared suite, checks the generation inputs and neighboring outputs, and rebuilds the scenario from a blank fictional contract. A second case uses a made-up reference but duplicates the production wording closely. It also needs review before publication. A simple pattern matcher can catch standard reference formats; it cannot judge whether an uncommon sequence of events is recognizable.
Fixture source: fictional rule schema; no raw customer transcript.
Exact scan: names, addresses, parcel references, account keys.
Similarity review: rare event sequence and distinctive wording.
If matched: quarantine case, inspect neighboring outputs, rebuild.Performance and operating cost
Scanning C outputs against R restricted identifiers takes O(CR) in a naive comparison, though indexed exact matches can reduce that work. Near-match and narrative review cost more and require access controls. Do not export raw comparison hits into broad build logs. The safest path for a small evaluation suite is often to author fresh fictional cases from policy states, avoiding production text entirely. When statistical fidelity is required, use a separately assessed data-generation process and measure utility and disclosure risk; this lesson's prompt workflow does not certify either.
Common Mistakes
- Do not assume replacing a name removes every identifying detail.
- Do not publish a generated case because its ID looks fictional.
- Do not describe prompt-generated fixtures as formally privacy-preserving data.
Connected lessons
- Production prompt engineering
- Prompt Engineering
- Prompt privacy: send only the fields needed for the task
- Prompt telemetry: measure failures without copying private payloads
- Evaluation leakage: keep the release test independent
- Synthetic evaluation cases: mutate a contract, not a customer's record
- Synthetic case oracles: reject invalid labels before scoring a model
- Synthetic case coverage: count distinct decisions, not rewritten sentences
- Synthetic evaluation holdouts: stop the generator from teaching the test
- Project: build a checked synthetic return-triage suite
- Synthetic evaluation-case decisions