Distributed model search: trial lineage and resource isolation

Last updated: 7 Oct 20265 min read

tutorial

A search result is useful only when each trial has a comparable dataset, split, metric and resource budget.

Freeze the search contract

A parcel-damage model search runs many candidates in parallel. Pin the training snapshot, feature code, split IDs, target definition, metric implementation and search-space revision before dispatch. Each trial receives a unique ID, parameter set, seed policy, image digest and resource request. A trial that reads a newer dataset is a separate experiment, even if its parameters look identical. Run manifests provide the artifact trail; snapshot replay keeps the comparison fair.

Separate workers and side effects

Trial workers should write outputs under unique IDs and never share a mutable checkpoint path. An interrupted worker may retry the same trial ID only under an idempotent claim, or it may create duplicate results that skew selection. Limit CPU, accelerator, memory and concurrent trials so the search cannot starve serving or other training jobs. Record queued, running, pruned, failed and completed states. Training admission supplies the larger resource boundary.

Keep selection data out of training

Fit preprocessing within each training fold and evaluate on its corresponding validation fold. Do not inspect the untouched final set after every trial or adapt the search space to it; repeated peeking turns it into a tuning set. Group or time boundaries must match deployment. A high validation score from leaked entity overlap is not a production gain. Feature parity and slice gates test the selected candidate after the search.

Promote the reproducible winner

Store the winner’s full trial lineage and rerun it from the pinned snapshot. Compare the rerun within a stated tolerance, then evaluate it once on the reserved final set and against the approved baseline. Report failed and pruned trials rather than only the winner. Pruning controls define fair early stops; the project catches two workers that accidentally share a checkpoint prefix.

Implementation

def trial_identity(trial, search_manifest):
    required = ("snapshot_digest", "split_digest", "metric_revision",
                "search_revision")
    for name in required:
        if trial.get(name) != search_manifest.get(name):
            return "reject:" + name
    if not trial.get("trial_id") or not trial.get("image_digest"):
        return "reject:identity"
    return "admit"

manifest = {"snapshot_digest": "parcel-s47", "split_digest": "group-r8",
            "metric_revision": "miss-cost-r3", "search_revision": "space-r4"}
trial = {**manifest, "trial_id": "parcel-trial-82", "image_digest": "image-r31"}
assert trial_identity(trial, manifest) == "admit"
assert trial_identity({**trial, "split_digest": "random-r7"},
                      manifest) == "reject:split_digest"

Performance and operating cost

The identity check is O(k) time and O(1) extra space for k pinned fields. The search itself costs the sum of trial training time and storage, plus scheduler contention. Strong resource isolation may leave some accelerators temporarily idle but prevents a broad search from delaying approved production work.

Common Mistakes

Comparing trials trained on different snapshots under one search ID.
Letting workers overwrite a shared checkpoint path.
Tuning against the final test set after each promising result.
Publishing only the best score while hiding trial failures and budget.