Project: combine warehouse pilots without double-counting evidence

Last updated: 7 Oct 20265 min read

project

Harmonize route-time effects, remove overlapping reports, and judge uncertainty for a proposed new warehouse.

Build a study inventory

A company has route-policy results from several warehouses. Record each site, dates, assignment, intervention version, eligible shipment class, effect scale, standard error, and report lineage. The release question is the effect of the current policy on mean scan time for a comparable new warehouse. One report uses minutes, another seconds, and a third is a revised analysis of the first pilot. Convert units only when the estimand matches, and keep one independent estimate per pilot. The alignment lesson supplies the first audit.

Inspect rather than average immediately

Plot or list site effects with their uncertainty and note which warehouse had a scanner upgrade during the pilot. Recalculate variances for cluster assignment where needed. If an observational site used policy A only on difficult routes, its contrast targets a different question from randomized sites unless adjustment assumptions are defended. Do not let a precise but incompatible result dominate the weight. Correct independent units and compatible estimands are prerequisites.

Report both mean and deployment risk

Compute a named average-effect analysis only for compatible studies, then examine between-site variation and a future-site range when the number of sites supports it. A favorable mean can coexist with a plausible slowdown at the next warehouse. Show how results change when the scanner-upgrade site is separated, and whether the proposed site resembles those studied. If few independent sites remain, frame the result as evidence for a monitored pilot rather than a guaranteed network-wide gain. Heterogeneity is decision information.

Publish an auditable packet

The packet includes the report-overlap map, unit conversions, effect definitions, variance corrections, included and excluded sites, chosen synthesis model, site-level results, and new-site caution. The gate below stops the synthesis before calculation if duplicate pilots or incompatible effects remain. A passing gate begins statistical review; it is not proof that transport or causal assumptions hold. Target-mix work may be needed for a site with different routes.

Implementation

def synthesis_gate(inventory, limits):
    if inventory["duplicate_pilots"]:
        return "hold:overlap"
    if inventory["unconverted_effect_scales"]:
        return "hold:effect-scale"
    if inventory["uncorrected_cluster_variances"]:
        return "hold:variance"
    if inventory["independent_sites"] < limits["minimum_sites"]:
        return "review:local-pilot-only"
    return "review:transport-assumptions"

limits = {"minimum_sites": 5}
inventory = {"duplicate_pilots": 1, "unconverted_effect_scales": 0,
             "uncorrected_cluster_variances": 0, "independent_sites": 6}
assert synthesis_gate(inventory, limits) == "hold:overlap"
assert synthesis_gate({**inventory, "duplicate_pilots": 0}, limits)        == "review:transport-assumptions"

Performance and operating cost

The gate is O(1). The study inventory is O(k) over reports, but overlap detection and correction of site variances can require source-data review. Pooling the already reported point estimates is cheap precisely because it ignores those failure boundaries.

Common Mistakes

Double-counting a revised report as another pilot.
Converting units while leaving effect definitions incompatible.
Using a precise average as a guarantee for a new warehouse.
Including biased observational contrasts without a stated transport argument.