Prior sensitivity: show when limited data leave a decision exposed

Last updated: 7 Oct 20265 min read

tutorial

A prior is part of a Bayesian model; compare defensible alternatives when its influence can change an operational decision.

Name the rate and time window

A helpdesk tracks the fraction of support tickets escalated within one day. Define which tickets count, when the outcome is mature and whether repeat contacts from one customer are independent. A beta-binomial model treats observed escalations as conditionally exchangeable Bernoulli outcomes with one common rate. That compact assumption can fail if queues or shifts differ. The target population determines what the rate represents.

Make the prior interpretable

Write a Beta prior with positive shape values before looking at this period. Its mean is alpha divided by alpha plus beta; that sum indicates how much prior concentration the model carries, although it is not automatically a literal sample of past tickets. After s escalations among n mature tickets, the posterior is Beta(alpha+s, beta+n-s). Compare a low-concentration prior with a prior reflecting documented past operations. The basic update supplies the arithmetic; the applied question is whether the chosen prior changes the action.

Test sensitivity at the decision boundary

Translate each posterior into the same decision rule, cost assumptions and observation window. If one reasonable prior favors intervention and another does not, report the decision as prior-sensitive rather than selecting whichever result is convenient. Check prior predictive implications: would this prior regularly predict implausible escalation counts before seeing current tickets? A prior with a plausible mean can still be too concentrated. Predictive checks inspect a different failure: whether the common-rate model can reproduce the observed group pattern.

Record the uncertainty that remains

A posterior interval answers a probability statement conditional on the model and prior, unlike a frequentist confidence interval. Neither removes biased ticket selection, delayed escalation labels or a changed queue policy. Archive the prior choices, mature outcome counts, posterior summaries and action thresholds. Decision probabilities connect uncertainty to the action; the project shows two priors leading to different recommendations.

Implementation

def escalation_posterior(prior_alpha, prior_beta, escalations, mature_tickets):
    if prior_alpha <= 0 or prior_beta <= 0:
        raise ValueError("prior shapes must be positive")
    if not 0 <= escalations <= mature_tickets:
        raise ValueError("invalid observed counts")
    alpha = prior_alpha + escalations
    beta = prior_beta + mature_tickets - escalations
    return {"alpha": alpha, "beta": beta, "mean": alpha / (alpha + beta)}

weak = escalation_posterior(2, 18, 9, 47)
skeptical = escalation_posterior(4, 56, 9, 47)
assert weak["mean"] > 0.14
assert skeptical["mean"] < 0.14

Performance and operating cost

A conjugate update is O(1) time and space after valid counts exist. Repeating it across k priors costs O(k), while collecting comparable, mature tickets and checking group structure dominate operational effort. More numerical precision cannot compensate for a prior chosen after seeing the preferred action.

Common Mistakes

Describing alpha plus beta as observed prior tickets without evidence.
Picking the prior that produces the desired escalation decision.
Treating a narrow posterior as protection against a changed ticket frame.
Pooling repeat contacts as independent Bernoulli trials without checking dependence.