Project: decide whether an invoice workflow preserves speed and safety

Last updated: 7 Oct 20265 min read

project

Run a two-endpoint release review with duration limits, a missed-duplicate harm margin and assignment-aware uncertainty.

Lock the pilot before comparison

A finance team tests a revised invoice-review screen across branches. The proposed screen reduces routine handling cost, but the release rule requires review duration to remain inside a predeclared two-sided tolerance and missed-duplicate rate to stay below a one-sided harm allowance. Record eligible invoices, branch assignment, reviewer IDs, start and stop timestamps, duplicate adjudication window and planned sample sizes. The endpoints use different units and cannot be collapsed into one favorable score. Duration equivalence and error noninferiority impose separate evidence requirements.

Find the apparent-equivalence trap

The first export includes only completed reviews. The new interface has more unresolved complex invoices, so observed duration looks close to the incumbent while those hard cases are absent. Rebuild the eligible frame, account for incomplete reviews and report outcome maturity by arm. Branch-level assignment also means individual invoice rows are not independent randomized units. The first narrow interval counted every invoice as a fresh assignment; it must be replaced with branch-aware uncertainty. Cluster planning explains why the number of branches matters.

Apply both boundaries without moving them

Compute a valid interval for candidate-minus-incumbent duration and require full containment inside the duration limits. Independently compute a bound on the missed-duplicate difference and require its upper edge below the permitted loss. Show assigned-group and adherence-sensitive estimates when staff switch screens. If those analyses disagree, identify the crossing pattern before claiming safety. Cost savings remain an additional result, not a way to offset a failed safety gate. Interval coverage depends on the extraction and grouping.

Release a reviewable decision packet

The packet includes signed contrast definitions, margins and their operating rationale, branch and invoice counts, completion flow, both intervals, tail duration, severe misses and the source-query version. The gate below refuses release when either endpoint fails or cases remain unevaluated. It accepts precomputed interval decisions; it does not generate evidence. A failed gate calls for additional follow-up or redesign, not a narrower margin selected from the same results. Additional outcomes and interim looks require their own plan.

Implementation

def invoice_release_gate(packet):
    if packet["unresolved_outcomes"] or packet["missing_branch_ids"]:
        return "hold:data-contract"
    if not packet["duration_interval_inside_margins"]:
        return "hold:duration-equivalence"
    if not packet["miss_rate_upper_bound_below_margin"]:
        return "hold:duplicate-safety"
    if not packet["assignment_and_adherence_agree"]:
        return "review:analysis-set-disagreement"
    return "review:eligible-for-release"

packet = {"unresolved_outcomes": 0, "missing_branch_ids": 0,
          "duration_interval_inside_margins": True,
          "miss_rate_upper_bound_below_margin": False,
          "assignment_and_adherence_agree": True}
assert invoice_release_gate(packet) == "hold:duplicate-safety"
assert invoice_release_gate({**packet,
       "miss_rate_upper_bound_below_margin": True}) == "review:eligible-for-release"

Performance and operating cost

The gate is O(1) time and space once interval results are audited. Building the packet is O(n) or more across invoices and branch clusters, and duplicate adjudication can dominate elapsed time. Reusing the completed-case export saves work only by silently changing the population being compared.

Common Mistakes

Publishing duration equivalence from completed cases alone.
Treating invoices as independently assigned when branches received the intervention.
Using cost savings to excuse a failed miss-rate safety bound.
Selecting the easier of two analyses after seeing results.