Leave-one-branch-out stability for rollout effects

Last updated: 5 Oct 20265 min read

tutorial

A leave-one-branch-out check recalculates a fixed rollout contrast after removing each entire branch to show whether one assignment unit determines the result.

Inspect influence at the treatment unit

A breach-rate effect may look stable across thousands of cases while resting on only three treated branches. If one branch had a system outage or a large staffing change, the pooled contrast may reflect that branch. Remove each whole branch in turn, recompute the treated-minus-control mean change, and compare the range with the full contrast. Do not remove isolated monthly rows from a branch; the treatment and correlated measurements belong together. Branch resampling uses the same unit for a different purpose.

Keep the estimand fixed as far as possible

This teaching check takes already selected treated and comparison branches and one pre-to-post change per branch. Removing a branch changes the finite set being averaged, so the result is an influence diagnostic rather than a new estimate of the original population effect. If a real design uses cohort weights, reweighting, or a comparison set selected from a network, state whether those steps are recomputed after each removal.

Read sign changes as a warning

The fixture has three treated and three control branches. Its full contrast is negative, and each leave-one-out estimate remains negative. That does not prove causal validity; a common trend violation could move every treated branch together. If the sign flips after deleting one branch, inspect its data lineage, baseline trajectory and assignment rationale before releasing a single summary. Record branch counts beside every estimate.

Avoid a false interval label

The minimum and maximum leave-one-out estimates are not a confidence interval. They have no automatic coverage guarantee and do not account for shared shocks across branches. With one treated branch, deleting it leaves no treated group, so this diagnostic is unavailable. A few-treated design may need assignment-based inference if a real lottery exists, or a clearly limited descriptive analysis if not. Assignment tests explain the boundary.

Connect outliers to operations

A branch with a sharp change may be a true part of the treatment effect or may have changed the measurement system. Do not delete it merely because it is influential. Compare incident logs, denominator changes and pre-rollout trends, then present the full result and sensitivity together. The review packet records any exclusion rationale.

Implementation

def leave_one_branch_out(treated_changes, control_changes):
    if len(treated_changes) < 2 or len(control_changes) < 2:
        raise ValueError("need at least two branches per group")
    if set(treated_changes) & set(control_changes):
        raise ValueError("branch appears in both groups")
    def contrast(treated, control):
        return (sum(treated.values()) / len(treated) -
                sum(control.values()) / len(control))
    full = contrast(treated_changes, control_changes)
    revisions = {}
    for branch in treated_changes:
        remaining = {key: value for key, value in treated_changes.items()
                     if key != branch}
        revisions[branch] = contrast(remaining, control_changes)
    for branch in control_changes:
        remaining = {key: value for key, value in control_changes.items()
                     if key != branch}
        revisions[branch] = contrast(treated_changes, remaining)
    return full, revisions

treated = {"North": -.06, "East": -.04, "Central": -.07}
comparison = {"West": .01, "South": .00, "Harbor": .02}
full_effect, without_branch = leave_one_branch_out(treated, comparison)
assert abs(full_effect - (-.06666666666666667)) < 1e-12
assert len(without_branch) == 6
assert all(value < 0 for value in without_branch.values())

Performance and operating cost

For B branches, the direct dictionary-copy implementation is O(B²) time and O(B) transient space per removal, plus O(B) output. Cached group sums reduce calculation to O(B) total time. Diagnostic design and branch-level data review dominate compute cost.

Common Mistakes

Do not label the leave-one-out range a confidence interval.
Do not drop an influential branch without an evidence-based exclusion rule.
Do not interpret stable signs as proof of parallel trends.