Combining studies: align effects, units, and independent evidence

Last updated: 7 Oct 20265 min read

tutorial

A weighted study summary is meaningful only when each input estimates a compatible effect on the same scale.

Define the study-level question

Several warehouses test a revised picking route, and each reports the difference in mean scan time between assigned policies. Before combining estimates, fix candidate minus incumbent, the time unit, eligible shipment type, follow-up window, and assignment unit. A study of median delivery delay or a ratio of failure risks is not interchangeable with a mean-time difference. Multiple reports from the same warehouse pilot may reuse participants and are not independent studies. The estimand lesson prevents an attractive numerical average of unlike quantities.

Use precision without erasing design

Under a common-effect model with independent, compatible estimates and known variances, each study contributes inverse-variance weight. More precise studies exert more influence. The code below computes only that weighted point estimate; it does not test whether the common-effect assumption is credible or calculate a confidence interval. A warehouse with thousands of correlated shipments can report a falsely tiny standard error if it ignores route-level assignment. Cluster-aware uncertainty must be fixed inside each study before synthesis.

Audit overlap and reporting selection

Build a registry of sites, dates, intervention versions, shipment eligibility, effect definitions, and raw data overlap. Count the same pilot once even if it appears in two reports. Look for completed but missing studies and unpublished unfavorable outcomes. A mathematically exact weighted mean cannot repair selective availability of results. Separate randomized from observational contrasts and explain why their estimands can or cannot be combined. The heterogeneity lesson addresses differences among underlying effects.

Report what the average can say

Show each study estimate and uncertainty, the harmonization decisions, and the weighted result under a named model. If effects vary materially by site, a common estimate may be too narrow a summary for a new warehouse. Report a new-site range or state that the available studies do not support transport. The project catches a duplicate pilot and a scale mismatch before either enters the release packet.

Implementation

def common_effect_mean(study_effects, study_variances):
    if not study_effects or len(study_effects) != len(study_variances):
        raise ValueError("aligned study estimates required")
    if any(variance <= 0 for variance in study_variances):
        raise ValueError("positive study variances required")
    weights = [1 / variance for variance in study_variances]
    return sum(effect * weight for effect, weight in
               zip(study_effects, weights)) / sum(weights)

combined = common_effect_mean([-1.2, -0.6], [0.16, 0.25])
assert round(combined, 6) == round(-9.9 / 10.25, 6)

Performance and operating cost

The weighted mean is O(k) time and O(k) space for k studies as written. Harmonizing outcomes, verifying non-overlap, and auditing each variance usually dominate effort. Adding more reports cheaply can add no independent information when they reuse the same pilot.

Common Mistakes

Pooling seconds with percentage changes without conversion and a shared estimand.
Counting two reports of one pilot as two independent studies.
Using row-independent standard errors from cluster-assigned warehouses.
Treating a common-effect average as a prediction for a new site.