Project: keep receipt scoring safe during feature publication lag

Last updated: 6 Oct 20265 min read

project

Test a delayed regional feature feed, preserve online event order and restore the scoring path through a staged repair.

Set the receipt policy

A receipt score uses merchant activity that must be no older than 47 minutes at decision time. The feature record carries event and availability timestamps, publisher revision and region. A missing or stale feature routes to manual review under a declared reason code. Freshness policy is checked per request even when the feature store reports that a record exists.

Build fault fixtures

Include current values, an entity whose value is 82 minutes old, a late event older than the online value, an equal-time correction with a higher revision, a future timestamp and a region whose materialization watermark stops. Freeze expected serve, fallback and publish decisions. Make the stopped region small enough that aggregate global metrics could hide it, forcing slice-aware investigation.

Stage recovery

Detect the stalled source or publisher using separate watermarks, replay the missing partition into a staging view and reconcile entity counts and event-time maxima. An older arriving event must remain historical rather than replacing current online state. Only then switch the affected region to the repaired view. Materialization repair keeps the scope bounded; parity checks compare repaired values with historical reconstruction.

Report customer outcomes

Measure stale-feature fallback rate, detection delay, repair duration, incorrectly overwritten entities and new receipt scores after recovery. Confirm no previously issued decision is silently rewritten. Keep sample receipt payloads restricted and alert by low-cardinality region and feature view. The drill is complete only when the scoring service resumes with fresh values and the fallback count returns to its expected range.

Implementation

from datetime import timedelta

def freshness_release(records, decision_at, maximum_age_minutes=47):
    failures = []
    for entity_id, row in records.items():
        if row["event_at"] > decision_at:
            failures.append((entity_id, "future"))
        elif decision_at - row["event_at"] > timedelta(
                minutes=maximum_age_minutes):
            failures.append((entity_id, "stale"))
    return {"state": "hold" if failures else "ready",
            "failures": failures}

from datetime import datetime, timezone
now = datetime(2026, 10, 6, tzinfo=timezone.utc)
records = {"merchant-47": {"event_at": now - timedelta(minutes=23)},
           "merchant-82": {"event_at": now - timedelta(minutes=82)}}
assert freshness_release(records, now)["state"] == "hold"
assert freshness_release({"merchant-47": records["merchant-47"]},
                         now)["state"] == "ready"

Performance and operating cost

Checking n entity records is O(n) time and O(f) space for f failures. A real release samples or batches checks across a large store and then enforces request-time freshness for every live decision. Staged repair doubles storage for the affected view until comparison and rollback windows close.

Common Mistakes

Assuming a global healthy watermark proves every region is current.
Letting a late event overwrite a newer online value.
Repairing the store while allowing stale features to score normally.
Rewriting previously issued decisions after a feature correction.