Prediction-time feature availability: reject future information before training

Last updated: 5 Oct 20265 min read

tutorial

A valid feature must be obtainable at the moment a prediction is made, with the same meaning in historical training and live serving.

Draw the prediction clock

A receipt team predicts whether a new submission will need manual review. The decision happens at submission time. Final reviewer notes, payment settlement status and a later correction timestamp are unavailable then, even if historical tables contain them. A model that uses these fields can score exceptionally in offline tests and fail immediately in use. List each candidate feature with its earliest availability time, source, update delay and owner. A pinned data snapshot] does not by itself prevent this temporal leak.

Join as of the decision

Build training rows from the state visible at each submission timestamp. An account summary should include only events known before that timestamp, with a documented lateness allowance; joining a current account table to past submissions rewrites history. For each derived feature, test a record whose outcome arrives after the decision and assert the value is unchanged. Join cardinality] also matters: a duplicated key can make the same outcome appear many times.

Audit serving parity

At deployment, compute the same feature names, units, null rules and ordering from available services. Log schema and feature-quality counters, not raw private values. A feature may be legally available but too slow or unreliable for the latency budget. If the online version falls back, the offline evaluation must include that fallback distribution. A single preprocessing pipeline] reduces transform drift; it cannot create data that has not arrived yet.

Challenge the suspicious winner

Remove each high-importance feature in turn and compare scores on a future or held-out group split. An abrupt collapse is a prompt to inspect availability and provenance, not proof of wrongdoing. Preserve the feature audit next to the model artifact so later source-table changes trigger review.

Implementation

feature_contract = {
    "account_age_days": "available_at_submission",
    "receipt_amount": "available_at_submission",
    "final_review_note": "after_outcome",
}
allowed_features = [name for name, availability in feature_contract.items()
                    if availability == "available_at_submission"]
assert "final_review_note" not in allowed_features
training_inputs = submission_snapshot[allowed_features]

Performance and operating cost

As-of joins can require indexed historical storage and cost O(N log M) for N decisions and M source events. The larger operational cost is maintaining feature freshness, backfills and fallback behavior across training and serving.