Covariate shift and support overlap

Last updated: 5 Oct 20265 min read

tutorial

Input mix can change while the outcome mechanism remains similar, but adapting to a new mix requires training support where deployment cases occur.

Start with the actual serving population

A shipment classifier was trained mostly on low-backlog intakes. A new depot contributes more high-backlog cases. Compare the feature distribution at the prediction timestamp, not a later enriched record. A changed marginal input distribution is evidence of input shift; it does not establish that the relation between input and outcome stayed fixed. Feature availability determines which columns can even be compared.

Make overlap visible

Bucket a decision-relevant variable and count training and current cases in each bucket. The code flags a bucket that appears in current traffic but has no training support. That is a coverage failure: reweighting old rows cannot teach the model how outcomes behave in a region with no examples. Multivariate overlap can fail even when every single feature has a familiar range, so inspect important combinations and distances as well. Local support is a complementary row-level test.

Do not confuse drift with damage

A large seasonal mix change can leave decisions sound; a small measurement change in one decisive feature can break them. Pair input counts with mature future error, calibration, review load and the existing baseline. If outcomes are delayed, document the gap between current input evidence and later performance evidence. Delayed-label monitoring defines the clock.

Define a response for missing support

Mark unsupported cases for a safe fallback or review, and collect labels under an explicit sampling plan. A model retrained on the same sparse region does not acquire new evidence. If a new depot uses a different target definition, that is more than covariate shift; reconcile the labels before pooling. The group audit checks whether one site bears most of the failures.

Keep the window honest

Compare like-for-like intake windows and account for known schedule cycles. Log the reference cohort, current cohort, bucket boundaries, missing-value rules and the model version. A threshold chosen after inspecting a bad production week is a development decision and needs a fresh evaluation window. The response project turns the diagnostic into a release decision.

Implementation

training_backlog = [1, 2, 2, 3, 4, 5, 6, 7]
current_backlog = [2, 4, 6, 9, 10, 11]

def backlog_band(queue_length):
    if queue_length < 4:
        return "low"
    if queue_length < 8:
        return "middle"
    return "high"

def band_counts(values):
    counts = {band: 0 for band in ("low", "middle", "high")}
    for queue_length in values:
        counts[backlog_band(queue_length)] += 1
    return counts

training_counts = band_counts(training_backlog)
current_counts = band_counts(current_backlog)
unsupported = [band for band, current_count in current_counts.items()
               if current_count and training_counts[band] == 0]
assert training_counts == {"low": 4, "middle": 4, "high": 0}
assert current_counts == {"low": 1, "middle": 2, "high": 3}
assert unsupported == ["high"]

Performance and operating cost

For N reference and M current rows, counting fixed bands costs O(N + M) time and O(B) memory for B bands. Multivariate support checks cost more and can become unreliable in sparse high-dimensional spaces; the cost of obtaining representative labeled cases is often the true constraint.

Common Mistakes

Do not infer stable outcomes from stable or changing feature histograms alone.
Do not extrapolate an importance weight into a bucket with zero training support.
Do not compare post-outcome features with prediction-time features.