Training resource gates: bound cost before starting a run

Last updated: 6 Oct 20265 min read

tutorial

A training request should state its compute budget, stopping conditions and recovery plan before it occupies shared capacity.

Estimate cost from a pilot

Before launching a long receipt model run, measure steps per second, accelerator memory and checkpoint time on a small representative slice. Estimate total compute hours and leave a margin for evaluation, restart and data loading. A pilot with short examples can understate cost for long-tail inputs, so include worst-case shapes. Record the estimate and observed spend against the same run ID. Checkpointing converts some interruption risk into storage and I/O cost.

Admit jobs under shared limits

A queue can reserve accelerator count, memory and maximum runtime per team. Reject or defer a job whose request exceeds its approved budget or whose data snapshot is missing. Do not let several high-priority experiments starve scheduled retraining. Distinguish exploratory runs from production candidates: a cheap exploratory run can skip full evaluation, but it cannot be promoted without the complete release evidence. Promotion gates remain separate from compute admission.

Stop unproductive work deliberately

Set a maximum wall clock, spend limit, failed-step count and early-stop rule before viewing results. Early stopping should use a development metric, not repeatedly peek at the final holdout. A run that stops for budget reasons is not a failed model evaluation; mark its artifact incomplete and unpromotable. If a preemptible machine saves money but causes repeated restarts, include lost work and delay when comparing it with stable capacity.

Audit the estimate

Compare predicted and actual runtime, checkpoint overhead, retries and evaluation cost. When a job exceeds its limit, preserve the last valid checkpoint and state why it stopped. The training recovery project simulates preemption and a budget breach, then asks whether the run should resume, restart or end.

Implementation

def admit_training(estimated_gpu_hours, budget_gpu_hours,
                   checkpoint_supported, expected_interruptions):
    if min(estimated_gpu_hours, budget_gpu_hours, expected_interruptions) < 0:
        raise ValueError("resource values must be nonnegative")
    if estimated_gpu_hours > budget_gpu_hours:
        return {"state": "defer", "reason": "budget"}
    if expected_interruptions > 0 and not checkpoint_supported:
        return {"state": "defer", "reason": "no-recovery"}
    return {"state": "admit", "reserved_gpu_hours": estimated_gpu_hours}

assert admit_training(18.5, 23, True, 2)["state"] == "admit"
assert admit_training(28, 23, True, 2)["reason"] == "budget"
assert admit_training(18.5, 23, False, 2)["reason"] == "no-recovery"

Performance and operating cost

Admission is O(1) time and space; estimating runtime requires a pilot or historical run data. Reserved capacity can remain idle if jobs start late, while underestimates can block other teams. Record measured cost after every run and revise estimates rather than turning the gate into an arbitrary fixed number.

Common Mistakes

Launching full training without a representative throughput pilot.
Calling an incomplete budget-stopped checkpoint a release candidate.
Using the final holdout for repeated early-stop decisions.
Ignoring restart time when claiming preemptible capacity is cheaper.