Geospatial evaluation: spatial holdouts, leakage and transfer

Last updated: 5 Oct 20265 min read

tutorial

Nearby records often share conditions, so a random row split can overstate how well a spatial model works in a new area.

Split by the deployment question

If a delivery-delay model will serve a new district, hold out entire districts or coherent geographic blocks. If it will predict future weeks in known districts, use a time split with enough gap to prevent late outcomes leaking. Group and time validation can be combined when both geography and time change at deployment.

Watch overlap and proxies

Adjacent cells may share one road closure, weather event or service center. A model that sees neighboring cells in training can memorize that event. Location IDs, postal codes and coordinates may act as proxies for the held-out area; decide whether those features will be available and meaningful in the new area. Keep a spatial buffer when the test requires independence from nearby training points.

Report coverage

Measure error by region, density, boundary distance and site age, with counts. A pooled score dominated by dense urban sites says little about remote locations. Track how often candidate retrieval excludes the true service point before evaluating travel-time ranking. Index freshness can affect both coverage and score.

Use a fixture

Create sites in three districts and deliveries over four weeks. Train on the first two districts and test on the third, then compare with a random-row split. A large random-split advantage should trigger an overlap investigation rather than a launch claim. Include a border site whose nearest service point lies in another district to test candidate eligibility.

Implementation

def held_out_regions(records, test_region_ids):
    test_regions = set(test_region_ids)
    train = [row for row in records if row["region_id"] not in test_regions]
    test = [row for row in records if row["region_id"] in test_regions]
    return train, test

Performance and operating cost

Partitioning N rows is O(N + R) expected time and O(N + R) memory for R held-out region IDs. Spatial buffer calculations and repeated folds add cost; that cost is justified when random splits answer the wrong deployment question.

Common Mistakes

Do not use a random row split for a new-region claim without an overlap check.
Do not report only dense-region performance.
Do not ignore true candidates missing from the spatial index.