Training and validation modes: measure the model you will serve

Last updated: 5 Oct 20265 min read

tutorial

Validation switches the model to evaluation behavior, disables gradient work and weights metrics by observed examples.

Separate optimization from measurement

Training updates parameters; validation estimates behavior on records that did not guide those updates. Set training mode before optimization and evaluation mode before measurement. Dropout and batch normalization change behavior across modes. A validation score computed in training mode measures a different system from the one served. Use inference mode during validation to avoid retaining a gradient graph.

Count images, not batches

The last batch may contain fewer images. Average loss weighted by batch size, then divide by the evaluated image count; an average of batch means overweights a short final batch. Track confusion counts by relevant group, not only rounded accuracy. Holdout design] determines whether the result answers the intended future question.

Freeze the selection rule

Choose a primary validation metric and checkpoint rule before inspecting results. Repeatedly trying architectures, thresholds and preprocessing against the same holdout adapts the project to that holdout. Keep a final untouched test set for one evaluation after selection. Augmentation and splitting] must respect this boundary.

Verify mode restoration

Run validation on two batches, including a short batch, and calculate the weighted mean by hand. Assert the model has a known mode after the helper returns. A validation helper that leaves the model in evaluation mode can silently disable training behavior in the next epoch.

Implementation

def validation_loss(receipt_model, validation_loader, loss_function):
    receipt_model.eval()
    weighted_loss = 0.0
    image_count = 0
    with torch.inference_mode():
        for images, class_ids in validation_loader:
            batch_loss = loss_function(receipt_model(images), class_ids)
            weighted_loss += batch_loss.item() * len(images)
            image_count += len(images)
    if image_count == 0:
        raise ValueError("empty validation set")
    return weighted_loss / image_count

Performance and operating cost

Validation takes O(V) forward evaluations for V examples without a backward graph. Frequent validation spends training time; infrequent validation can miss overfitting or an unstable update.

Common Mistakes

Do not evaluate dropout or batch normalization in training mode.
Do not average batch means without weighting the short batch.
Do not tune repeatedly on the final test partition.