Project: measure support-handoff clarity with a versioned survey

Last updated: 7 Oct 20265 min read

project

Create a survey audit that preserves the case frame, item meaning, branch states and form-version comparability.

Build the invitation frame

Select 53 eligible support cases with a handoff in the previous 14 days, one contact per case. Preserve invitation ID, case ID, handoff date, queue, form assignment and delivery status. Record exclusions before invitation. The response file contains 31 completed surveys and 22 nonresponses; those counts must reconcile to the eligible frame. Frame checks determine who could be represented.

Specify and pilot the measure

Ask one neutral question about how clear the next step was after the named handoff. Use ordered clarity labels, plus explicit “do not remember” and a not-displayed state for the no-handoff branch. Run a nine-person cognitive pilot that includes both handoff and no-handoff experiences. Log ambiguous interpretations and branch defects, revise, and retest. Pilot checks produce a release decision, not a population estimate.

Reconcile response states

Suppose 27 respondents saw the item and 23 gave a substantive clarity answer; 17 of those chose the top two categories. Report 17/23 as the top-two share among substantive item answers, alongside 23/27 substantive completion and 31/53 survey completion. Do not bury four non-substantive item responses in the top-two denominator. The codebook keeps skip, non-display and recall failure distinct.

Run a version overlap

Assign 27 invitations to form A and 26 to form B before fielding. In the fixture, A yields 16 completed responses and B yields 15. Keep exact wording, order and option IDs for both forms. Compare invitation, response and item-answer counts before comparing their clarity distributions. The version lesson explains why respondent-only differences are not automatically wording effects.

Deliver an honest report

Publish the survey contract, frame counts, pilot issue log, form version map, branch-validity results, category distribution, respondent-only estimate and nonresponse caveat. Investigate whether nonrespondents differ on known case attributes. If a form revision is not comparable, mark the trend break and keep both versions in the export. Nonresponse follow-up is a separate analysis task.

Implementation

def reconcile_survey(invitation_ids, completed_ids, substantive_answers):
    invitations = set(invitation_ids)
    completions = set(completed_ids)
    if len(invitations) != len(invitation_ids):
        raise ValueError("duplicate invitation IDs")
    if len(completions) != len(completed_ids) or not completions <= invitations:
        raise ValueError("invalid completed response IDs")
    if not 0 <= substantive_answers <= len(completions):
        raise ValueError("invalid substantive answer count")
    return {"invited": len(invitations), "completed": len(completions),
            "nonresponse": len(invitations - completions),
            "substantive_answers": substantive_answers}

packet = reconcile_survey(range(53), range(31), 23)
assert packet == {"invited": 53, "completed": 31,
                  "nonresponse": 22, "substantive_answers": 23}

Performance and operating cost

Set reconciliation costs O(I + R) time and space for I invitations and R response IDs. Category summaries are O(R). The costly step is establishing what the question measures and whether the respondent frame supports the intended claim.

Common Mistakes

Do not label respondent-only clarity as the opinion of all 53 invitees.
Do not merge form versions without a comparability check.
Do not delete invalid branch records before recording the defect rate.