Project: govern a distributed parcel-damage model search

Last updated: 7 Oct 20265 min read

project

Run a bounded search, reject incomparable trials and evaluate the winner without leaking the final holdout.

Freeze the parcel problem

The task predicts whether a parcel image needs manual damage review. Group images by shipment so frames from one parcel never appear in both training and validation. Pin the snapshot, group split, feature preprocessing, miss-cost metric, search space and worker image. Set a budget of 82 trials and a finite accelerator-hour cap before dispatch. Trial identity rejects workers pointed at a newer image archive or a random split.

Exercise worker failure

Launch workers with unique trial and checkpoint prefixes. Two accidentally share a prefix; the second resumes a first trial’s checkpoint and emits a plausible metric. Detect the mismatch through trial ID and checkpoint metadata, quarantine both outputs and rerun them cleanly. Another worker reports at step 39 while peers report at step 82; it cannot be pruned by direct score comparison. Comparable milestones define the stop rule.

Select once, then evaluate

At the budget cap, freeze the completed trial list and rerun the chosen configuration from the pinned snapshot. Inspect damage types, camera classes and shipment groups. The top validation candidate improves the global miss-cost objective but loses on crushed-corner parcels; hold it pending a documented quality decision rather than using the untouched final set to pick a different trial. After selection is fixed, evaluate once on the protected final set and record the result without further search.

Deliver the release packet

Publish trial counts by completed, pruned, failed and quarantined states; total compute, winner lineage, rerun tolerance, slice report and serving-resource estimate. A later search needs a new search revision and its own final evaluation plan. Registry gates should receive the selected artifact only after data, performance and slice checks pass. The project is not complete merely because a scheduler returned a best-trial number.

Implementation

def parcel_search_disposition(trials, budget, selected_trial):
    if budget <= 0 or len(trials) > budget:
        return "hold:budget"
    valid = [trial for trial in trials if trial["state"] == "completed"
             and trial["lineage_ok"]]
    selected = next((trial for trial in valid
                     if trial["trial_id"] == selected_trial), None)
    if selected is None:
        return "hold:invalid-winner"
    if not selected["slice_gate_passed"]:
        return "hold:slice-quality"
    return "evaluate:final-holdout"

trials = [{"trial_id": "parcel-47", "state": "completed",
           "lineage_ok": True, "slice_gate_passed": False},
          {"trial_id": "parcel-82", "state": "completed",
           "lineage_ok": True, "slice_gate_passed": True}]
assert parcel_search_disposition(trials, 82, "parcel-47") == "hold:slice-quality"
assert parcel_search_disposition(trials, 82, "parcel-82") ==        "evaluate:final-holdout"

Performance and operating cost

The gate is O(t) time and O(t) temporary space for t trial records. Training dominates cost; the fixed budget makes it visible before results tempt expansion. Retrying corrupted trials adds compute, but accepting mixed checkpoints would make the comparison invalid and could waste a later production rollout.

Common Mistakes

Sharing checkpoint paths between concurrent trials.
Pruning a trial from a metric reported at an earlier step.
Using the final holdout to select among the top search trials.
Ignoring rare-damage failures because the global objective improved.