Project: audit a rare duplicate-payment rule before replacing it

Last updated: 7 Oct 20265 min read

project

Compare two claim controls with reconciled denominators, a declared effect scale, and a sparse-cell uncertainty review.

Define the policy decision

A claims team proposes a new duplicate-payment flag. Decide whether the target is confirmed duplicate payments per eligible claim, false flags per reviewed claim, or total review cost; these are distinct outcomes. Freeze the approval window and ensure both policies had a comparable opportunity to detect an event. Match the unit of assignment, the unit counted, and the interval used for uncertainty. The estimand lesson defines the target population.

Reconcile the audit trail

Construct a four-cell table for each chosen binary outcome, then verify every claim ID appears once in the analysis unit and that policy labels match the actual rule running at decision time. Repeated flags on one account may demand account-level treatment. A zero or small cell is visible evidence, not an invitation to suppress the table. The exact-test lesson provides one prespecified one-sided calculation when its fixed-margin model fits.

Quantify business effect and uncertainty

Give candidate and baseline risks, absolute difference, and ratio only if the baseline risk is positive. Convert the absolute difference to expected events for the planned volume while stating that transport requires comparable future claims and detection effort. Add a suitable sparse-event interval and a review of costs from false flags. If event adjudication is blinded differently across policies, a statistical test does not repair that measurement difference. The effect-scale lesson prevents odds and risks from being mixed.

Set the release gate

The packet includes policy version, eligible counts, detected and adjudicated events, repeated-account handling, exact or interval method, false-flag burden, and the operational threshold. The gate below halts release for irreconcilable counts or missing decisions about the independent unit. Passing it only allows review of effect and uncertainty; it does not assert that a new rule is better. Preserve disagreement cases for adjudicator recheck and monitor for drift after rollout.

Implementation

def claims_rule_gate(audit):
    if audit["candidate_events"] > audit["candidate_total"] or        audit["baseline_events"] > audit["baseline_total"]:
        return "hold:count-reconciliation"
    if not audit["independent_unit_defined"]:
        return "hold:independent-unit"
    if not audit["matched_observation_window"]:
        return "hold:window"
    return "review:effect-and-cost"

audit = {"candidate_events": 3, "candidate_total": 240,
         "baseline_events": 9, "baseline_total": 240,
         "independent_unit_defined": False, "matched_observation_window": True}
assert claims_rule_gate(audit) == "hold:independent-unit"
assert claims_rule_gate({**audit, "independent_unit_defined": True})        == "review:effect-and-cost"

Performance and operating cost

The gate and point comparisons are O(1). Claim-level reconciliation is O(n) over records when IDs are indexed. The expensive work is outcome adjudication and correcting repeated-account dependence; no faster test substitutes for a trustworthy four-cell table.

Common Mistakes

Mixing confirmed duplicates with unadjudicated flags.
Treating repeated flags on one claim as independent events.
Comparing unequal detection windows.
Approving a rule from a favorable ratio while ignoring absolute benefit and false-flag cost.