Project: audit privacy exposure in a return-review model

Last updated: 7 Oct 20265 min read

project

Match audit cohorts, limit client output fields and hold a model change that harms rare-return review.

Inventory what clients can see

A return-review service sends route and reason to warehouse scanners, but an older API revision also exposes a high-precision probability. Pin model digest, response scope and caller class. Create a permitted audit with known training and nontraining returns from the same period and comparable channels. Keep account-group boundaries intact. The audit frame is limited to outputs a given caller could actually receive.

Catch a confounded result

The first diagnostic compares training records from one seasonal promotion with holdout records from another. Its large confidence gap looks alarming, but the periods have different return patterns. Mark that run incomparable. Rebuild matched cohorts, freeze a threshold before scoring and report true-positive and false-positive rates, counts and the resulting gap. A low gap is not a blanket privacy certificate.

Test mitigation and task utility

Remove probability from the scanner response while retaining it only for the approved risk desk. A separately trained candidate has a lower measured exposure gap but misses rare damaged-item returns that the incumbent flags. Hold that candidate. The release gate requires both privacy evidence and protected task quality. Validate risk-desk contracts before changing its output scope.

Publish a controlled packet

Record cohort definitions, output access, threshold, model revisions, diagnostic rates, task slices, caller migrations and canary owner. Keep record-level audit guesses out of broad dashboards. If a later candidate passes, canary the scoped response and monitor client failures; investigate any residual concerns without presenting one diagnostic as proof. Connect the packet to log retention and registry review.

Implementation

def privacy_case_disposition(audit, candidate, limits):
    if not audit["cohorts_matched"] or not audit["threshold_predeclared"]:
        return "hold:audit-design"
    if candidate["rare_damage_recall"] < limits["rare_damage_recall"]:
        return "hold:task-quality"
    if audit["rate_gap"] > limits["rate_gap"]:
        return "review:exposure"
    return "canary:scoped-client"

limits = {"rare_damage_recall": 0.82, "rate_gap": 0.18}
audit = {"cohorts_matched": True, "threshold_predeclared": True,
         "rate_gap": 0.11}
candidate = {"rare_damage_recall": 0.76}
assert privacy_case_disposition(audit, candidate, limits) == "hold:task-quality"
assert privacy_case_disposition(audit, {"rare_damage_recall": 0.87},
                                limits) == "canary:scoped-client"

Performance and operating cost

The gate is O(1) time and space after reports are prepared. Matched audit data, human-reviewed rare-return labels and client contract tests are the expensive parts. Restricting the scanner response reduces one exposure surface without requiring the team to discard a useful incumbent model prematurely.

Common Mistakes

Calling a season difference evidence of membership leakage.
Publishing per-record membership guesses to a general dashboard.
Removing probabilities from a client that is approved to use them.
Promoting a candidate with weaker rare-damage recall because one audit gap fell.