Project: release an evidence-backed lesson-reminder policy

Specify the intervention

At 09:00, a learner who is eligible and has not completed the current lesson may receive one reminder. The primary outcome is completion within seven days; opt-out and complaint events are guardrails. Freeze the reminder text, delivery channel, time zone and seven-day window. The target contract determines which decision episodes enter analysis.

Prepare evidence in order

Audit one row per decision and verify logged assignment probability. Keep a randomized holdout with both arms in every planned target segment. Wait for outcome maturity, split by learner and calendar time, fit S- and T-learner baselines, then fit a cross-fitted augmented weighting model only if its added complexity improves held-out policy value. Baselines and cross-fitting answer different failure modes.

Review support and uncertainty

Report propensity distribution, arm counts and effective sample size for high- and low-score bands. On the untouched randomized period, compare the proposed top-capacity policy with no reminders and with the prior uniform policy. Show effect uncertainty by learner-level resampling; a positive point estimate whose interval includes material loss is not a release case. Overlap checks can remove unsupported segments before any rollout.

Replay the actual serving rule

Apply opt-outs, contact cooldown, positive net value and a daily cap of 24 in the same order in offline evaluation and production. Keep a randomized measurement lane. Record model version, eligibility version, decision timestamp, action probability, chosen action and reason. Capacity-constrained policy design explains the order; policy-value evaluation measures the complete rule.

Set a release and rollback gate

The packet below represents a failed overlap check, so it blocks release despite positive estimated value. The human review should also inspect assignment-ratio defects, immature outcomes, opt-out spikes and any complaint harm. During the ramp, stop new reminders if the log pipeline fails or a guardrail breaches its declared threshold; restore the previous policy and retain the randomized comparison for diagnosis.

Implementation

release_packet = {
    "assignment_log_complete": True,
    "outcome_window_mature": True,
    "randomization_check_passed": True,
    "overlap_check_passed": False,
    "incremental_completions_lower_bound": 2.4,
    "daily_contact_capacity": 24,
    "planned_daily_contacts": 19,
    "opt_out_guardrail_passed": True,
}

def review_reminder_release(packet):
    failures = []
    for check in ("assignment_log_complete", "outcome_window_mature",
                  "randomization_check_passed", "overlap_check_passed",
                  "opt_out_guardrail_passed"):
        if not packet[check]:
            failures.append(check)
    if packet["incremental_completions_lower_bound"] <= 0:
        failures.append("incremental_value_uncertain")
    if packet["planned_daily_contacts"] > packet["daily_contact_capacity"]:
        failures.append("capacity_exceeded")
    return {"release": not failures, "failures": failures}

assert review_reminder_release(release_packet) == {
    "release": False, "failures": ["overlap_check_passed"]}

Performance and operating cost

The packet check is O(1); producing trustworthy fields requires event reconciliation, repeated nuisance fitting, grouped uncertainty estimates and live guardrail monitoring. Randomized holdout consumes contact opportunity but measures whether the policy still helps after user behavior and content change.

Common Mistakes

Do not approve from a positive point estimate when support or uncertainty fails.
Do not drop the control arm during rollout.
Do not claim a live gain from an offline score without checking the served eligibility and capacity rules.