Experiment contract: decision, estimand and assignment unit

Last updated: 5 Oct 20265 min read

tutorial

An online experiment estimates an effect for a declared population, outcome and assignment unit; changing any one changes the question.

Write the release question

AI Trove could compare two lesson recommendation policies for active learners. The decision is whether to ship the new policy; the primary outcome might be a qualifying lesson completion within seven days of assignment. Specify whether the effect is among all eligible learners or only people who opened the feed. The population and estimand must precede a dashboard query.

Choose an independent unit

Assign by stable learner ID if the same learner can return on several devices. Randomizing each page view causes a person to encounter both variants and changes the estimand to a page-view effect. If a classroom shares lessons and decisions, classroom assignment may be safer, but cluster correlation reduces effective sample size. Keep the unit, salt, allocation and assignment version in an immutable manifest.

Define eligibility before treatment

A learner is eligible based on facts known before policy exposure: account state, locale and a stable consent rule. Do not exclude learners after assignment because the new feed loaded slowly or returned an empty slate. Those failures are part of the effect. Record assignment once, then log every later eligibility or suppression decision separately so the intention-to-treat population stays recoverable.

Exercise the boundary

A learner enters on a phone and returns on a laptop; both requests should receive the same variant. A second learner with no eligible lessons still belongs to the assigned population if eligibility was declared at account level. The test packet should make clear whether this empty slate counts as a poor experience, a technical failure or a predeclared exclusion.

Implementation

import hashlib

def assign_variant(learner_id, experiment_salt, treatment_share=0.5):
    digest = hashlib.sha256(f"{experiment_salt}:{learner_id}".encode()).digest()
    bucket = int.from_bytes(digest[:8], "big") / (1 << 64)
    return "treatment" if bucket < treatment_share else "control"

Performance and operating cost

Hash assignment is O(L) time for an ID and salt of length L and O(1) retained state. Stable hashing avoids a lookup on each request, but changing the salt or ID mapping midrun invalidates assignment continuity.

Common Mistakes

Do not randomize page views when the effect is defined per learner.
Do not remove post-assignment failures from the primary population.
Do not change the allocation salt without recording a new experiment version.