Experiment telemetry: assignment, exposure and outcome events

Last updated: 6 Oct 20265 min read

tutorial

Assignment says who was placed in a variant; exposure says who could encounter it; outcome events say what happened afterward.

Keep three event families

Store assignment with learner, experiment, variant and version. Emit exposure only when the policy actually renders a meaningful feed, and write completion outcomes under their own event ID and event time. A retry may resend any event. Idempotent loading should collapse retries without erasing legitimate repeated activity.

Preserve the primary analysis

If the treatment fails before rendering, that failure must not disappear from the assigned-population estimate. Exposure-only analysis answers a narrower question and can be biased because exposure depends on the variant. Report both assignment and exposure counts, with reasons for nonexposure. A server-side flag read is not proof a learner saw the recommendation.

Make joins time-aware

A completion before assignment cannot be an outcome of the assigned variant. Decide whether a qualifying completion requires a lesson from the feed or any lesson; the former may reflect ranking, the latter overall learning activity. Use event identity, learner identity and an explicit attribution horizon. Late mobile uploads may revise counts, so freeze a maturity cutoff for final reporting.

Test duplicate and delayed data

Create one assignment, one rendered feed, two deliveries of the same completion event and another completion uploaded after the seven-day reporting cutoff. The duplicate must count once; the late upload may revise a provisional result under the documented correction rule. A treatment rendering failure must remain in the assignment table even when no exposure exists.

Implementation

def unique_qualifying_events(events, assigned_at, horizon_end):
    seen_ids = set()
    qualifying = []
    for event in events:
        if event["event_id"] in seen_ids:
            continue
        seen_ids.add(event["event_id"])
        if assigned_at <= event["occurred_at"] < horizon_end:
            qualifying.append(event)
    return qualifying

Performance and operating cost

A pass over N events is O(N) expected time and O(U + Q) memory for U event IDs and Q qualifying events. At scale, durable event keys and partitioned time windows replace an unbounded in-memory set.

Common Mistakes

Do not equate a feature-flag read with real exposure.
Do not drop failed renders from intention-to-treat analysis.
Do not count retried outcome events twice.