Prior sensitivity for an operational action threshold

Last updated: 5 Oct 20265 min read

tutorial

A decision supported by a small audit should be recalculated under plausible prior strengths before it changes staffing.

Hold the data and action fixed

The audit has 11 escalations among 53 mature cases. Leadership plans extra review capacity when evidence suggests the underlying rate exceeds 17%. Fix that rate threshold and the loss of an unnecessary shift before comparing priors. A prior-sensitivity check changes only the prior assumptions, not the observed cohort or decision rule. The decision objective defines what an extra shift is meant to prevent.

Compare equal-mean priors

Beta(1, 9), Beta(4, 36) and Beta(12, 108) all have prior mean 10% but concentrations of 10, 40 and 120. With the same 11 events in 53 cases, posterior means are 12/63, about 19.0%; 15/93, about 16.1%; and 23/173, about 13.3%. The 17% action threshold lies inside this spread. A decision based only on one prior mean or one posterior mean would conceal that fragility.

Check plausibility before preference

The concentrated prior might reflect a stable historical service process, or it might be stale after a queue redesign. Review the older cohort, its outcome definition and its prior predictive implications. Prior predictive checks can reject implausible choices before the current audit. Do not pick the weak prior just because it triggers the desired staffing action, or the strong prior because it avoids cost.

Compare decision quantities

For each prior, calculate the posterior probability that the rate exceeds 17%, not just the mean. A cost-sensitive policy may require probability above a prespecified level. Show the probability range and whether all plausible priors lead to the same action. If they disagree, either collect more mature cases or use a conservative temporary action whose cost is acceptable. Posterior decisions are conditional on the model.

Do not use sensitivity as a ritual

A list of arbitrary priors has little value. Tie alternatives to distinct evidence states: little reliable history, a comparable prior period, and a strong but potentially outdated history. Save parameters and rationale. If the likelihood is wrong because West and East have different rates, prior sensitivity alone will not repair that structural problem. Group checks target the likelihood.

Implementation

def posterior_means_by_prior(prior_shapes, escalations, audited):
    if not 0 <= escalations <= audited:
        raise ValueError("invalid audit counts")
    means = {}
    for prior_name, (alpha, beta) in prior_shapes.items():
        if alpha <= 0 or beta <= 0:
            raise ValueError("prior shapes must be positive")
        means[prior_name] = (alpha + escalations) / (alpha + beta + audited)
    return means

priors = {"weak": (1, 9), "working": (4, 36), "strong": (12, 108)}
means = posterior_means_by_prior(priors, 11, 53)
assert round(means["weak"], 3) == 0.190
assert round(means["working"], 3) == 0.161
assert round(means["strong"], 3) == 0.133

Performance and operating cost

Computing posterior means for P candidate priors is O(P) time and output space. Probability calculations using Monte Carlo add sampling cost per prior. More computation cannot identify which prior population is comparable; that requires data lineage and domain review.

Common Mistakes

Do not choose a prior after seeing which staffing action it favors.
Do not vary the prior while silently changing the audit cohort or threshold.
Do not use prior sensitivity to excuse a poor likelihood model.