Project: audit a depot scan trial with blocked assignment and shared staff

Last updated: 7 Oct 20265 min read

project

Reconstruct the shift lottery, check contamination, and decide whether a blocked randomization analysis supports deployment.

Freeze design and target

A logistics team tested a new scan sequence on selected shifts inside each depot. Define eligible shifts, depot blocks, the fixed number assigned to the new process in each block, the outcome measured per shift, and the target policy effect. Preserve the assignment ledger even when a shift has no usable scan-time measure. A row-level analysis of individual parcels would create false precision because the lottery happened at shift level. The assignment lesson identifies the unit.

Rebuild the actual lottery

Check that every recorded treatment label is possible under the block-specific allocation and that the analysis does not move labels across depots. Freeze one shift-level statistic and a one-sided or two-sided decision before examining results. Enumerate the within-block assignments for a small study or sample assignments from the same lottery for a larger one. The randomization lesson demonstrates the sharp-null test, whose p-value is not a policy effect estimate.

Measure operational spillover

Join shifts to staff handoffs, scanner queues, and route transfers. Count control shifts with treated neighbors and note when configuration persisted across the boundary. If interference is substantial, revise the estimand and study design before claiming an isolated shift effect; consider depot-week assignment for a follow-up trial. The interference lesson explains the tradeoff between containment and independent cluster count.

Release the evidence packet

The packet holds the lottery seed or allocation log, block sizes, missing-outcome ledger, prespecified statistic, effect estimate and uncertainty, permutation result, spillover map, and an operational cost threshold. The gate below blocks claims when assignments do not match the planned lottery or shared-staff exposure has not been audited. A passing gate permits a scoped result, not an assertion that every depot will improve. Monitor the implemented rollout against the trial conditions.

Implementation

def depot_trial_gate(audit):
    if audit["assignment_violations"]:
        return "hold:lottery-mismatch"
    if audit["missing_shift_outcomes"] and not audit["missingness_reviewed"]:
        return "hold:missing-shifts"
    if not audit["spillover_audited"]:
        return "hold:shared-staff-exposure"
    if not audit["statistic_predeclared"]:
        return "hold:analysis-selection"
    return "review:scoped-depot-effect"

audit = {"assignment_violations": 0, "missing_shift_outcomes": 1,
         "missingness_reviewed": True, "spillover_audited": False,
         "statistic_predeclared": True}
assert depot_trial_gate(audit) == "hold:shared-staff-exposure"
assert depot_trial_gate({**audit, "spillover_audited": True}) ==        "review:scoped-depot-effect"

Performance and operating cost

The gate is O(1). Exact assignment enumeration can grow combinatorially with shifts per block; the staff-sharing audit is O(V + E) after roster extraction. Treating every parcel as an independent replicate is cheap but gives the wrong uncertainty for a shift lottery.

Common Mistakes

Permuting across depots despite within-depot assignment.
Dropping shifts with missing outcomes after seeing their labels.
Ignoring scanner and staff spillover into controls.
Using a sharp-null p-value as the size of the rollout benefit.