Human overrides: keep decisions, reasons and labels distinct

Last updated: 7 Oct 20265 min read

tutorial

A reviewer may override a model route, but that action is not automatically a ground-truth label or a model error.

Record three separate events

Store the model score and proposed route, the reviewer action, and the eventual adjudicated outcome as separate events tied to one decision ID. The reviewer may have information the model never saw, or may make a mistake. Treating every override as a positive label contaminates training and evaluation. Keep reviewer identity under restricted access, but preserve role, reason code, timestamp and policy revision for audit. Decision logging supplies a privacy-aware join; the label ledger records later corrections.

Design reasons that support investigation

A short controlled set of reason codes should distinguish missing feature, document quality, contextual evidence, policy exception and reviewer disagreement. Free text may help an investigation but can capture sensitive customer information; apply access and retention rules. Do not force a reviewer to choose “model wrong” when the reason was an expired feature. Review the codes periodically for ambiguity. Admission failures should be visible before the model score so the reviewer is not blamed for an input problem.

Measure override rates with denominators

Report overrides per eligible decision and per reviewed decision, broken down by model digest, policy revision, reason and meaningful cohort. A rising override count could reflect a lower review threshold, a staffing shift or real quality loss. Compare the same eligibility window and expose missing reviewer records. An override rate alone is not a calibration score; use mature outcomes to decide whether model or policy changes improve decisions. Threshold changes can alter which cases reach humans.

Close the feedback loop deliberately

Create a queue for adjudicating disputed decisions, with an owner, evidence standard and correction history. Only adjudicated outcomes may enter the evaluation set under its documented label rule. Feed recurring reason codes into a feature or policy investigation, not directly into automatic retraining. The project injects a reviewer override caused by a stale feature and tests whether the system keeps that reason separate from model accuracy. The record should allow an operator to reconstruct what each actor knew at decision time.

Implementation

def append_review_event(history, event):
    if event["decision_id"] not in history:
        history[event["decision_id"]] = []
    if event["kind"] not in {"model", "reviewer", "adjudication"}:
        raise ValueError("unknown event kind")
    history[event["decision_id"]].append(event)
    return history[event["decision_id"]]

record = {}
append_review_event(record, {"decision_id": "receipt-82", "kind": "model",
                             "route": "clear"})
append_review_event(record, {"decision_id": "receipt-82", "kind": "reviewer",
                             "route": "review", "reason": "stale-feature"})
assert len(record["receipt-82"]) == 2
assert all(event["kind"] != "adjudication" for event in record["receipt-82"])

Performance and operating cost

Appending an event is O(1) expected time with O(e) retained space for e events. Production storage needs durable writes, access controls and deduplication by event ID. The small example separates event types but does not prove that a reviewer action was justified; adjudication and mature outcome evidence remain independent costs.

Common Mistakes

Using reviewer override as ground truth without adjudication.
Dropping the original model route after a human changes it.
Comparing override counts without stable denominators.
Using unrestricted free text as the only reason field.