Review a fictional Aster Console signup experiment with two assigned groups of 4,800 users each. The prompt may draft analysis from a locked packet, but it must preserve the randomization unit, outcome window, logging checks, and planned decision rule. A descriptive difference is not a license to ship while exposure records and a support-ticket guardrail remain unresolved.
Make this comfortable
Project: review Aster’s signup experiment
Review the packet
Control has 2,304 signup completions within 24 hours, or 48% of 4,800 assigned users. Treatment has 2,496, or 52%; the observed difference is 4 percentage points. The exposure ledger contains 4,600 control and 4,500 treatment records, fewer than assigned in both groups, so the team investigates logging and eligibility before interpreting the effect. Support-ticket counts are 96/4,800 (2%) in control and 144/4,800 (3%) in treatment. The packet contains no checked uncertainty estimate or confirmed reason for the logging gap. The current decision is hold, repair the evidence path, and rerun the locked analysis.
Assigned: control 4,800; treatment 4,800
24h completions: 2,304/4,800 = 48%; 2,496/4,800 = 52%
Observed difference: +4 percentage points
Exposure records: 4,600 control; 4,500 treatment; investigate
Support tickets: 96/4,800 = 2%; 144/4,800 = 3%
Decision: hold pending data-quality and guardrail reviewPerformance and review cost
Reconstructing N assignment and outcome records is O(N) expected time with an identity map; auditing several metrics and slices multiplies review work. More prose cannot compensate for a broken exposure ledger or an unplanned decision threshold. Store the randomization key, query version, observation cutoff, and investigation outcome so a later run can reproduce the same comparison.
Common Mistakes
- Do not call a four-point observed difference a proven improvement without validated inference.
- Do not change the denominator from assigned users to logged exposures mid-analysis.
- Do not hide the higher support-ticket rate when discussing rollout.
Related lessons
- Experiment prompts: fix hypothesis, unit, and exposure
- Experiment prompts: investigate assignment and exposure imbalance
- Experiment prompts: freeze the primary metric and guardrails
- Experiment prompts: reconcile missing outcomes by assigned unit
- Experiment prompts: separate effect size from uncertainty
- Experiment prompts: investigate slices and guardrail movement
- Experiment prompts: gate rollout on a reproducible packet
- Product experiment prompt decisions
- Evaluation sets: measure the failure cases that matter
- Report prompts: reconcile every claim with the packet
prompt engineering
product experimentation