Skip to content
AITroveRead. Build. Understand.
Reading list
Make this comfortable

Study effects: calculate the measure outside the model

Last updated: 4 Oct 202611 min read
tutorial
AdvancedBy AITrove Editorial

An effect-measure prompt states which event and direction matter, then hands arithmetic to deterministic code. For a binary outcome, each arm's observed event rate is events divided by its stated denominator. The candidate-minus-control risk difference is a signed absolute difference; multiply by 100 only when reporting percentage points. It is not a relative percent change or a confidence interval. Check that counts are integers, denominators are positive, events do not exceed denominators, and the time window is the same. The model may explain the result after the program computes it, but it must not turn a point estimate into evidence of statistical certainty or operational causation.

Operational case

In ST-47, the control rate is 48/240=20.0% and the candidate rate is 30/240=12.5%, so the candidate-minus-control difference is -7.5 percentage points. In ST-88, 18/120=15.0% and 21/120=17.5%, giving +2.5 points. One trial favors the candidate; the other does not. The assistant initially writes that the panel 'cuts misroutes by 7.5%' for all sorting lines, which confuses percentage points with relative percent and drops the contrary trial. The corrected report names each study's observed rates and leaves uncertainty and generalization for separate assessment.

python
studies = [
    {"id": "ST-47", "control_events": 48, "control_total": 240,
     "candidate_events": 30, "candidate_total": 240},
    {"id": "ST-88", "control_events": 18, "control_total": 120,
     "candidate_events": 21, "candidate_total": 120},
]

for study in studies:
    control_events = study["control_events"]
    control_total = study["control_total"]
    candidate_events = study["candidate_events"]
    candidate_total = study["candidate_total"]
    if not all(type(value) is int for value in (
        control_events, control_total, candidate_events, candidate_total
    )) or not (0 <= control_events <= control_total and control_total > 0
              and 0 <= candidate_events <= candidate_total and candidate_total > 0):
        raise ValueError(f"invalid event counts: {study['id']}")
    difference_points = 100 * (
        candidate_events / candidate_total - control_events / control_total
    )
    print(f"{study['id']}: {difference_points:+.2f} percentage points")

Performance and operating cost

Computing N independent study differences takes O(N) time and O(1) working space per streamed study. The code below validates the four counts in each record and prints signed differences. It does not adjust for cluster allocation, missing outcomes, study design, uncertainty, or dependencies among reports. Those choices require a separate analysis plan and suitable methods. Preserve raw counts alongside formatted percentages; rounding each arm before subtracting can produce a different displayed difference near a reporting threshold.

Common Mistakes

  • Do not confuse percentage points with relative percent change.
  • Do not infer certainty or causation from a point estimate alone.
  • Do not pool studies just because both have numeric outcomes.

Connected lessons

Continue with: Analytics prompts: explain segment gaps without inventing causes.

Continue with: Experiment prompts: separate effect size from uncertainty.

prompt engineering
research synthesis
Storage details