Difference-in-differences: compare changes under a defended trend assumption

Last updated: 5 Oct 20265 min read

tutorial

A two-group, two-period difference-in-differences contrast subtracts the control change from the treated change.

State the comparison

Imagine the panel launches in one academy but not another. Record the launch date, affected learners, outcome window and unit. If completion rises from 0.30 to 0.43 in the treated academy and from 0.28 to 0.34 in control, the two-period contrast is 0.07, or seven percentage points. That number is causal only under assumptions, especially that untreated trends would otherwise have evolved similarly.

Inspect earlier periods

Plot several prelaunch periods using the same outcome definition and cohort rule. Divergent earlier movement weakens the parallel-trends story; similar earlier movement supports scrutiny but cannot prove the unobserved postlaunch counterfactual. Calendar and missing-period rules must be shared across groups. A concurrent curriculum overhaul in one academy can break the comparison.

Watch changing composition

A surge of new learners after launch changes the treated group even if the panel has no effect. Report cohort mix, attrition and eligibility before and after in both academies. If treatment starts at different times across many groups, a simple two-period formula or unexamined pooled regression may target a different weighted effect. Keep the lesson’s formula to its stated design.

Challenge the design

Construct a placebo launch date before the real one and calculate the same contrast. A large placebo movement should prompt investigation. Also test a group unaffected by the panel but exposed to the same tracking changes. Placebo checks are evidence about plausibility, not proof of parallel trends.

Implementation

def two_period_difference_in_differences(treated_before, treated_after, control_before, control_after):
    rates = (treated_before, treated_after, control_before, control_after)
    if any(not 0 <= rate <= 1 for rate in rates):
        raise ValueError("completion rates must be between zero and one")
    return (treated_after - treated_before) - (control_after - control_before)

assert round(two_period_difference_in_differences(0.30, 0.43, 0.28, 0.34), 2) == 0.07

Performance and operating cost

The four-rate contrast is O(1) after group-period aggregation; building rates from N learner records is O(N). Computation is not the main limit: parallel trends, stable composition and concurrent changes determine whether the contrast is interpretable.

Common Mistakes

Do not treat a pretrend plot as proof of the unobserved counterfactual.
Do not ignore group composition changes after launch.
Do not apply the two-period formula to staggered adoption without revisiting the estimand.