Analytics prompts: explain segment gaps without inventing causes

Last updated: 4 Oct 202611 min read

tutorial

A segment comparison needs the same eligibility, identity, event definition, and observation window on both sides. Ask the model to report counts and rates, identify possible measurement or composition differences, and propose targeted checks. A descriptive gap is not proof of a design defect or of the effect of a planned fix. Seasonality, device mix, acquisition channels, missing events, and changed instrumentation are plausible alternatives. The memo should rank next tests by information value and leave the causal field unresolved until stronger evidence exists.

Operational case

Meridian's mobile signup rate is 1,175/2,350, or 50%; desktop is 1,645/2,350, or 70%. The gap is 20 percentage points. The analyst can investigate mobile form errors, client event loss, traffic sources, and version exposure, but the numbers alone do not show which mechanism is responsible. A product proposal to shorten the mobile form is a testable hypothesis, not a result already supported by this table.

Mobile: 1,175 / 2,350 = 50%
Desktop: 1,645 / 2,350 = 70%
Difference: 20 percentage points
Observed: segment gap
Unknown: cause and effect of a proposed form change

Performance and review cost

Comparing two grouped rates is O(1) arithmetic after an O(N) data preparation pass. Investigating confounders costs additional slices, logging checks, and often a controlled study. Keep a compact hypothesis ledger so the model does not repeat a single attractive explanation as fact across successive reports.

Common Mistakes

Do not call the 20-point gap a proven mobile form defect.
Do not hide the two 2,350-person denominators.
Do not compare segments built with different identity rules.

Connected lessons

Continue with: Experiment prompts: investigate slices and guardrail movement.

prompt engineering

product analytics