Buffered spatial holdout: test prediction away from nearby training sites

Last updated: 7 Oct 20265 min read

tutorial

A geographic test region plus a gap between it and training data can reveal performance hidden by nearby training neighbors.

Match the intended prediction

A retailer forecasts demand for branches in a new district. Randomly holding out rows from existing districts mostly tests interpolation between familiar neighbors. Instead, mark a geographic test region and exclude a surrounding buffer from training so near-border sites do not leak local conditions into the test. Coordinates must be projected into suitable distance units; raw longitude and latitude are not kilometers. The code uses planar kilometer coordinates and returns separate train, buffer and test sets. The neighbor lesson explains why this separation matters.

Keep preprocessing inside each split

If a regional average, target encoding, imputer or scaling parameter uses all branches before the split, the held-out geography has already influenced training. Fit each transformation on training only, then apply it to the held-out region. Repeat across several regions and report performance by region, branch size and demand range. A single easy district can make a model appear portable when remote districts fail. Time-ordered validation may be needed too when deployment is both in a new place and a later month.

Choose buffer size deliberately

The gap should reflect the range over which local conditions or shared records make sites similar, as well as the distance at which the business will deploy. Too narrow a gap leaves leakage; too wide a gap can remove most training examples and test an unrealistic extrapolation problem. Compare a few prespecified distances, record how many sites fall in each band, and show the same geography on a map. Do not tune the gap to obtain a desired error score.

Do not overstate the holdout

Spatial separation helps estimate performance for unsampled regions resembling the selected test regions. It does not make biased branch placement representative of every possible region, and it does not establish a causal effect of location. Report coordinate quality, site duplicates, overlapping catchments, fold assignment, buffer removals and deployment similarity. The project holds a release if the headline score came from random rows or leaked preprocessing.

Implementation

from math import hypot

def buffered_region_split(branch_rows, center_km, test_radius_km, buffer_km):
    if test_radius_km <= 0 or buffer_km < 0:
        raise ValueError("valid test radius and buffer required")
    train, gap, test = [], [], []
    for branch_id, x_km, y_km in branch_rows:
        if not branch_id:
            raise ValueError("branch ID required")
        distance = hypot(x_km - center_km[0], y_km - center_km[1])
        if distance <= test_radius_km:
            test.append(branch_id)
        elif distance <= test_radius_km + buffer_km:
            gap.append(branch_id)
        else:
            train.append(branch_id)
    if not train or not test:
        raise ValueError("split needs both training and test branches")
    return train, gap, test

train, gap, test = buffered_region_split([
    ("Hill", 0, 0), ("Market", 6, 0), ("Coast", 19, 0)], (0, 0), 4, 5)
assert (train, gap, test) == (["Coast"], ["Market"], ["Hill"])

Performance and operating cost

Partitioning n projected branch coordinates costs O(n) time and O(n) output space. Repeated model fits across geographic folds dominate compute. The buffer deliberately discards some training rows; that cost buys a test closer to deployment beyond a familiar neighborhood.

Common Mistakes

Computing kilometer buffers directly from unprojected degree coordinates.
Fitting preprocessing on all branches before geographic splitting.
Reporting random-row accuracy as new-region accuracy.
Selecting the buffer width after seeing which value looks best.