Project: operate receipt scoring with a deadline and overload path

Last updated: 7 Oct 20265 min read

project

Define a measurable receipt-scoring service objective, then test capacity, timeout and fallback behavior under failure.

Write the customer-visible contract

The service receives a receipt and either returns a risk decision within 240 ms or routes it to a declared manual-review state. Choose an end-to-end availability definition that counts timeout and invalid responses, not just process health. Define a target window, traffic population and exclusions before looking at a favorable dashboard. The latency budget allocates time across lookup, queue, model and response.

Build load and fault fixtures

Replay a normal hour, a short 3× burst, a slow feature-store period and a cold model restart. Include large payloads and a candidate model with higher CPU demand. Record accepted, timed out, overloaded and manual-review outcomes. Use the same inputs for the current and candidate release; a result on idle hardware is not comparable to one on a saturated node. Overload policy decides when the queue must stop growing.

Gate deployment on complete evidence

Require the candidate to meet quality and latency bounds while preserving enough rollback capacity. Check that the fallback path works when the feature store is unavailable, and that shadow requests are canceled when the main request ends. Emit low-cardinality duration metrics and trace identifiers; keep receipt payloads under a separate privacy policy. Canary a small traffic share and halt if customer-visible manual review exceeds its budget. Rollback must restore both model pointer and serving capacity.

Review the failure drill

Report end-to-end deadline success, tail latency, timeout rate, queue depth, fallback volume and cost per accepted score. During a simulated outage, verify that no fabricated safe score appears and that operations can distinguish capacity overload from malformed features. Store the test hardware, model digest and source revision so a later comparison is meaningful. A release is ready only when the rollback drill and the customer fallback meet their stated limits.

Implementation

def serving_gate(outcomes, minimum_total=470):
    total = sum(outcomes.values())
    if total < minimum_total:
        return {"state": "hold", "reason": "insufficient-traffic"}
    timely = outcomes.get("scored-in-budget", 0)
    unsafe = outcomes.get("fabricated-safe-score", 0)
    fallback = outcomes.get("manual-review", 0)
    if unsafe or timely / total < 0.96 or fallback / total > 0.04:
        return {"state": "hold", "reason": "service-objective"}
    return {"state": "ready", "evaluated": total}

window = {"scored-in-budget": 481, "manual-review": 19}
assert serving_gate(window)["state"] == "ready"
assert serving_gate({**window, "fabricated-safe-score": 1})["state"] == "hold"

Performance and operating cost

Aggregating k outcome counters takes O(k) time and O(1) auxiliary space. A load test costs compute and may need isolated capacity to avoid affecting customers. This gate checks one window only; production release policy should also cover traffic slices, tail latency and repeated windows so a brief good period cannot mask a persistent problem.

Common Mistakes

Measuring only model compute time instead of the request deadline.
Calling a process healthy while customers receive timeouts.
Sending shadow load without reserving capacity.
Treating manual review as free or unlimited capacity.