Project: release a receipt threshold with review evidence

Last updated: 6 Oct 20265 min read

project

Change a receipt review threshold under a fixed model digest, measure capacity and preserve human overrides without relabeling them.

Freeze the comparison

Use one approved receipt model digest and two policy revisions. The current rule reviews at score 0.81; the candidate reviews at 0.72. Specify tie behavior, eligible population, fallback route and reviewer capacity before replaying outcomes. Split policy selection from final evaluation by time and keep mature-label coverage visible. Threshold governance needs both action impact and quality evidence; the model card prevents expansion into unsupported uses.

Replay the decision set

Evaluate the two rules on the same frozen receipts and count review volume, missed urgent cases and slice-level changes. Include a new-merchant slice whose mature labels are sparse. Mark it insufficient rather than declaring the new threshold safe there. Simulate an afternoon burst to test whether the reviewer queue can absorb extra cases. Record the policy revision with each replayed route; otherwise a later model change can make the comparison impossible to explain.

Exercise the reviewer path

For one cleared receipt, a reviewer sees a stale feature and changes the route. Store the proposed route, override event and reason separately; do not add a positive label. Later adjudication may confirm or reject the concern, and its corrected outcome belongs in the label ledger. The override contract lets an operator separate missing input evidence from a model mistake. Force a duplicate reviewer submission and verify that the event ID prevents a second action.

Make a reversible decision

A staged release can stop when manual-review load crosses its approved limit, even if early quality numbers look promising. Deliver the fixed digest, old and candidate policies, cohort counts, mature-label coverage, capacity test, override examples, stop conditions and rollback pointer. Record whether broad release is approved, restricted or held. Live experiment exposure can follow the offline gate, but it must not bypass it. The final record states what remains unknown and who owns the next review.

Implementation

def policy_replay(scores, current_at, candidate_at, daily_review_limit):
    current = sum(score >= current_at for score in scores)
    candidate = sum(score >= candidate_at for score in scores)
    return {"current_review": current, "candidate_review": candidate,
            "state": "hold" if candidate > daily_review_limit else "pilot"}

receipt_scores = [0.12, 0.47, 0.73, 0.82, 0.94]
result = policy_replay(receipt_scores, 0.81, 0.72, 2)
assert result == {"current_review": 2, "candidate_review": 3,
                  "state": "hold"}
assert policy_replay(receipt_scores, 0.81, 0.72, 3)["state"] == "pilot"

Performance and operating cost

A replay over n scores is O(n) time and O(1) extra space for two thresholds. Full evaluation also joins labels, applies slice gates and simulates queue capacity; those steps consume substantially more time and evidence. The five scores are a deterministic illustration, not an estimate of a real queue or event rate.

Common Mistakes

Treating a lower threshold as a model release with no policy record.
Counting a reviewer override as an outcome label.
Approving the candidate before checking review capacity.
Discarding insufficient slice evidence because the aggregate improves.