Experiment prompts: investigate assignment and exposure imbalance

Last updated: 4 Oct 202611 min read

tutorial

The prompt should request the configured allocation, unique assigned-unit counts, identity exclusions, and exposure-log counts for each arm. An apparent sample-ratio mismatch is a data-quality symptom; it is not automatically a product effect or a verdict about randomization. Use a prespecified statistical check where the experiment platform supports one, but first inspect duplicate IDs, bot filters, missing events, and variant-specific logging. Do not declare a formal mismatch from two numbers without the allocation rule and check result. Keep the investigation outcome attached to the analysis packet.

Operational case

Aster intended a 1:1 assignment and observes exactly 4,800 unique assigned users in each arm. That count alone does not validate exposure logging: the recorded exposures are 4,600 and 4,500. The model lists plausible failure paths, such as variant-specific event loss or different eligibility at the surface. An engineer checks the emitters and the analyst verifies identity filters. Until the discrepancy is explained, the effect memo remains provisional.

Configured assignment 1:1
Observed assigned 4,800 : 4,800
Logged exposure 4,600 : 4,500
Status: investigate missing or unequal exposure records
Do not label a formal SRM without the planned check

Performance and review cost

Counting N unique assignments and exposures with hashed IDs is O(N) expected time and O(N) memory. Additional joins across client and server logs can dominate the investigation cost. Automated alerts are useful triage, but a reviewer must establish whether the mismatch comes from assignment, filtering, or telemetry before interpreting the outcome table.

Common Mistakes

Do not read the outcome delta before resolving a serious data-quality flag.
Do not treat equal assigned counts as proof every user saw the intended surface.
Do not call an unequal log count a formal statistical mismatch without its test.

Connected lessons

prompt engineering

product experimentation