Survey prompts: code open text without exposing respondents

Last updated: 4 Oct 202611 min read

tutorial

Open-text survey answers can contain names, addresses, account details, or stories about other people. Screen and minimize that content before model analysis, then keep pseudonymous response IDs and item versions attached to each retained excerpt. Use a codebook with inclusion and exclusion rules rather than allowing the model to invent a polished theme from a handful of vivid comments. Track how many distinct respondents contributed to each code, separate blank answers from branch skips, and retain contrary comments. A public report should not reproduce a rare identifying quote just because it is memorable.

Operational case

One Meridian response names a depot worker while describing a pickup delay. The analyst removes the name and keeps the operational detail under response R-47. Another answer says the broad window is useful because someone is home all day; it is a counterexample to the narrow-window proposal. The codebook marks planning difficulty and preference for flexibility separately. The final memo can summarize both without exposing the worker or pretending that one quote speaks for all 65 eligible respondents.

R-47: worker name removed; delay detail retained
Code PLANNING_DIFFICULTY: reviewed inclusion rule
Code FLEXIBILITY_PREFERENCE: contrary case retained
Count unit: distinct respondents, not repeated phrases
Public quote: suppress if re-identification risk remains

Performance and review cost

Screening T free-text answers and assigning K candidate codes directly is O(TK); deduplicating by respondent is O(T) expected time. Human review is needed for uncertain redactions, code changes, and rare quotations. A smaller, traceable excerpt set often gives a better analysis than sending every raw answer into a model and hoping the output stays private.

Common Mistakes

Do not put direct identifiers into the analysis batch without need.
Do not count repeated phrases as separate respondents.
Do not omit comments that challenge the leading interpretation.

Connected lessons

prompt engineering

survey research