Proxy measurements and label error in operational datasets

Last updated: 5 Oct 20265 min read

tutorial

A recorded field may be a convenient proxy for the decision quantity rather than a faithful measurement of it.

Name the hidden quantity

A support team wants to measure whether a customer issue was solved. The ticket field marked closed is easy to query, but closure may mean an agent sent a reply, not that the customer confirmed a fix. Declare the intended outcome and the recorded proxy separately. A good database join cannot make a poor proxy valid. One row per case is necessary, but it does not settle what each field means.

Audit disagreement with independent review

Draw a review sample from the full eligible frame, including open and closed cases. Reviewers inspect source conversation and later contact without seeing the proxy label where possible. Build a two-by-two count of proxy positive or negative against reviewed positive or negative, plus unknown review outcomes. If 47 reviewed cases contain 29 proxy-closed cases and only 22 genuinely solved cases, the difference in totals alone cannot tell which individual cases were misclassified; retain the paired labels.

Look for differential error

A closure field may be more accurate for email than chat, or for low-priority cases than urgent cases. Compare disagreement by channel, agent workflow and week. If the error rate changes at the same time as a product change, a trend in the proxy can masquerade as a trend in real outcomes. A reviewer can also err, so document adjudication and any later outcome evidence. Disagreement review is part of the measurement contract.

Bound what can be claimed

Report the raw proxy rate, the paired audit counts and the estimated reviewed rate under the sampling design. Avoid silently substituting one for another. If a rare segment has only three reviewed cases, its apparent correction factor is unstable. A population adjustment based on an audit needs valid inclusion probabilities and a defensible error model; without those, present a sensitivity range. Design weights matter when audit strata were sampled unequally.

Choose a repairable process

Sometimes a better event can be captured prospectively: confirmed fix, no repeat contact within a declared window, or a customer response. Each alternative has its own missingness and delay. Version the outcome rule and run old and new definitions in parallel before comparing periods. A metric definition change is a measurement change, not automatically an improvement in service.

Implementation

def paired_label_counts(reviewed_cases):
    counts = {"true_positive": 0, "false_positive": 0,
              "false_negative": 0, "true_negative": 0, "unknown": 0}
    for case in reviewed_cases:
        reviewed = case["reviewed_solved"]
        if reviewed is None:
            counts["unknown"] += 1
            continue
        proxy = bool(case["ticket_closed"])
        key = ("true_positive" if proxy else "false_negative") if reviewed else ("false_positive" if proxy else "true_negative")
        counts[key] += 1
    return counts

Performance and operating cost

Pairing N reviewed cases costs O(N) time and O(1) auxiliary space after identities are joined. The audit itself costs reviewer time and may require delayed follow-up; disagreement by K strata needs O(K) additional counters and enough cases per stratum to be informative.

Common Mistakes

Do not treat a convenient status field as the outcome without an audit.
Do not discard unknown reviewed outcomes from the audit ledger.
Do not apply one correction factor across channels when error rates may differ.