Two coders may read an interview segment differently for sound reasons. Capture each independent label, its supporting words, the codebook version, and the reason for disagreement before discussion. A model suggestion is another proposal, not an authoritative tie breaker. Adjudicate by checking the full segment and the code definition; if the evidence cannot settle the question, mark it unresolved and avoid a confident thematic claim. Report agreement only for an explicitly defined unit and label set. Do not turn a high agreement number into proof that the codes are meaningful or that the sample represents a wider population.
Make this comfortable
Interview prompts: keep coder disagreement visible
Operational case
Coder A marks P03-S18 as WINDOW_CLARITY because the revised time surprised the participant. Coder B marks it as NOTIFICATION_TIMING because the participant describes learning of the change too late. The full segment supports both mechanisms, so the review assigns two codes with separate evidence spans. Another unclear segment remains unresolved pending audio. The review log keeps the original assignments, the final decision, and who made it; a model does not erase the disagreement.
P03-S18 | codebook v2
Coder A: WINDOW_CLARITY | span 12:14-12:23
Coder B: NOTIFICATION_TIMING | span 12:25-12:42
Review: both codes, separate spans
P09-S04: unresolved until audio checkPerformance and review cost
A second independent pass roughly doubles coding time for the reviewed subset, and adjudication grows with disputed segments D rather than every segment N. Sample a meaningful range of cases, including near misses, instead of only easy examples. The audit trail adds a small amount of storage and prevents silent consensus. It also reveals whether a codebook revision, rather than a different model prompt, is the right fix.
Common Mistakes
- Do not let one model output overwrite independent coder decisions.
- Do not force a single label when distinct supported mechanisms coexist.
- Do not present unresolved segments as agreed evidence.
Connected lessons
- Production prompt engineering
- Prompt Engineering
- Interview prompts: write a codebook with inclusion rules
- Evaluation labels: adjudicate disagreement before scoring a release
- Interview prompts: limit transcript use to the study purpose
- Interview prompts: preserve speaker and segment boundaries
- Interview prompts: build a theme evidence ledger
- Interview prompts: count people, not repeated excerpts
- Interview prompts: turn bounded findings into testable decisions
- Project: analyze Parcel Window pickup interviews
- Qualitative interview prompt decisions
Continue with: Annotation prompts: measure agreement before adjudication.
prompt engineering
qualitative research