Project: review the effect of a new lesson recommendation panel

Freeze the question

Define the learner as assignment unit, the panel version as treatment and seven-day lesson completion as outcome. State the eligible population, follow-up cutoff, interference risk and primary intention-to-treat estimand. Build a fixture with assigned treatment and control learners, exposure flags, prior activity and missing outcomes. The target effect must be fixed before reading the completion counts.

Audit assignment and outcomes

Check duplicate learner keys, group counts, exposure imbalance and outcome collection by assigned group. Compute the completion-rate difference with all observed assigned units and give bounds for unknown outcomes. Do not quietly analyze only users who clicked the panel. Noncompliance belongs in a separate report.

Compare weaker designs

Construct a naive exposed-versus-unexposed estimate and a simple before-after estimate from the same fixture. Explain how motivation and time can move them away from the randomized contrast. If using a difference-in-differences comparison, inspect prelaunch trends and concurrent changes rather than treating its formula as automatic causal proof.

Submit an evidence packet

Include the assignment manifest, timestamped covariates, exclusion ledger, count reconciliation, estimate and uncertainty method, placebo or negative-control check, and a short release decision. A reviewer should reproduce every numerator and denominator. Report where evidence ends: a result for eligible learners this month does not automatically transfer to every learner or later catalog version.

Implementation

def assigned_completion_difference(treatment, control):
    if not treatment or not control:
        raise ValueError("both assigned groups are required")
    if any(outcome not in (0, 1) for outcome in treatment + control):
        raise ValueError("resolve or bound missing outcomes separately")
    return sum(treatment) / len(treatment) - sum(control) / len(control)

assert assigned_completion_difference([1, 0, 1], [0, 0, 1]) == 1 / 3

Performance and operating cost

Computing the observed contrast is O(N) time. Reconciliation and clustered uncertainty can require additional passes or resampling; preserve the immutable assignment ledger so later corrections do not rewrite which group a learner belonged to.

Common Mistakes

Do not substitute a clicker-only comparison for the assigned-group effect.
Do not use post-treatment clicks as baseline adjustment variables.
Do not publish a causal claim without a stated identification design and missing-outcome account.