Uplift targets and randomized event contracts

Last updated: 6 Oct 20265 min read

tutorial

Uplift estimates the change in an outcome caused by assigning an action, conditional on information available before that assignment.

Define the decision before the model

A learning platform can send a reminder to a learner who has not finished a lesson. The decision is whether to send it at 09:00 on an eligible day; the outcome is lesson completion within the next seven days. Sending at noon, sending a different message, or counting completion over 30 days defines another treatment question. The causal estimand fixes the unit, action, comparison and horizon before features are built.

Separate baseline response from incremental response

A learner with an 82% completion probability after a reminder may have completed with 80% probability anyway. The estimated uplift is two percentage points, not 82. Another learner with a 36% treated probability and 19% untreated probability has lower absolute response but 17 points of uplift. A policy that targets the first learner because the treated probability is highest can waste a scarce outreach slot.

Log the assignment mechanism

For every eligible decision, record learner ID, decision timestamp, assigned action, probability of that assignment, policy version, eligibility rule and pre-decision features. Keep rows for control assignments and unanswered messages. Assignment probability must be known at decision time. Experiment exposure logs help distinguish assignment from delivery; the primary intention-to-treat analysis uses assignment.

Preserve outcome maturity

A seven-day outcome is not available two days after assignment. Either wait for the complete window or mark the row immature; do not turn it into a non-completion. A learner can receive multiple reminders, so choose a washout rule or model a sequential policy instead of treating overlapping decisions as independent. Group-and-time validation keeps a learner from appearing on both sides of a test split.

Name the identification boundary

Random assignment makes treatment groups comparable in expectation; it does not repair missing outcomes, broken randomization, interference between learners, or a change in message content. Observational logs require measured confounders and overlap, with stronger assumptions than a randomized holdout. Overlap diagnostics test whether the proposed targeting population has support.

Implementation

decision_log = [
    {"learner": "acct-47", "assigned_at": 120, "action": 1, "assignment_probability": 0.46, "observed_through": 128},
    {"learner": "acct-83", "assigned_at": 124, "action": 0, "assignment_probability": 0.54, "observed_through": 126},
]

def mature_assignment_rows(records, outcome_window_days=7):
    mature = []
    for record in records:
        if record["action"] not in (0, 1):
            raise ValueError("unknown action")
        if not 0 < record["assignment_probability"] <= 1:
            raise ValueError("invalid logged assignment probability")
        if record["observed_through"] >= record["assigned_at"] + outcome_window_days:
            mature.append(record)
    return mature

assert [row["learner"] for row in mature_assignment_rows(decision_log)] == ["acct-47"]

Performance and operating cost

Filtering N decisions is O(N) time and O(N) output space. The costly part is a trustworthy event stream, a stable randomization key and seven-day outcome reconciliation. Retaining a permanent control fraction has opportunity cost, but without support for both actions a personalized causal claim cannot be checked.

Common Mistakes

Do not train on only delivered reminders when the decision was assignment.
Do not mark a two-day-old seven-day outcome as a failure.
Do not confuse high treated response with large treatment effect.