A prompt can create a review checklist, but it cannot substitute for people interpreting the survey in context. Pilot with a small group resembling the intended respondents, ask what each question meant to them, and inspect actual branch behavior. Log every issue with the item ID, observed interpretation, severity, proposed repair, owner, and retest result. Once fielding begins, freeze the question and branch version for that wave. An emergency fix may be necessary, but the analysis must mark which respondents saw each version and avoid pooling changed items without a compatibility argument.
Make this comfortable
Survey prompts: pilot the instrument before fielding
Operational case
Five Meridian pilot participants see Q7-v1. Two interpret 'pickup window' as the period for collecting a parcel from a locker, while the study means the delivery slot before pickup. The team rewrites the item as Q7-v2, tests it again, and fixes the cannot-recall branch found in the pilot. The 240-person invitation uses only the frozen v2 instrument. If a form fix is made mid-wave, its release time and affected response IDs are recorded rather than hidden in a final percentage.
Pilot: 5 people; 2 misread Q7-v1
Repair: clarify pre-confirmation time window in Q7-v2
Branch repair: cannot recall skips planning item
Retest: affected paths checked
Fielded instrument: frozen v2Performance and review cost
A pilot adds fixed recruitment and review time, while graph regression remains O(V+E) for V items and E branches. That cost is smaller than collecting a full wave whose key item cannot be interpreted. Store instrument checks as executable cases where possible, but preserve the participants' explanations too; a branch test cannot detect a word that readers understand differently.
Common Mistakes
- Do not treat a model's wording preference as respondent validation.
- Do not silently edit a fielded item.
- Do not close a pilot issue before the changed path is retested.
Connected lessons
- Production prompt engineering
- Prompt Engineering
- Evaluation sets: measure the failure cases that matter
- Prompt changes in CI: test the merge candidate
- Survey prompts: define the population and invitation frame
- Survey prompts: check wording against the intended construct
- Survey prompts: validate answer options and skip paths
- Survey prompts: reconcile invitations, submissions, and item denominators
- Survey prompts: code open text without exposing respondents
- Survey prompts: release findings with explicit scope
- Project: review Meridian pickup survey findings
- Survey prompt decisions
prompt engineering
survey research