Prior predictive checks for a binary service rate

Last updated: 5 Oct 20265 min read

tutorial

A prior is a probability model written before the current outcomes are seen; test the outcomes it implies before updating it.

Define the event and sampling unit

A support audit labels a checked case as escalated or not escalated within seven days. One case contributes one binary outcome, and the parameter is the escalation probability for the eligible case population during that period. A beta prior represents uncertainty about that probability. It does not fix the probability for every customer segment or solve an incomplete sampling frame. Coverage checks come first.

Choose parameters on the outcome scale

Suppose an earlier, comparable operating period suggests a rate near 10%. Beta(3, 27) has mean 3/30, or 10%, and concentration 30. That concentration affects how strongly the prior influences small samples; it is not a claim that exactly 30 historical cases existed. Record the origin of this belief, population match and review date. A prior borrowed from a different queue can be systematically wrong.

Simulate the audit before it happens

For each possible rate drawn from the prior, simulate the number of escalations in a 47-case audit. Inspect the range and extremes. A prior that frequently predicts implausible audit counts needs revision before looking at the current labels. The exact probability of zero escalations under this prior is roughly 5.2%. That calculation is a prior predictive probability, not the chance that the true rate is zero.

Check competing priors

Compare a concentrated Beta(3, 27) with a weaker prior that has the same mean, such as Beta(1, 9). They predict the same average rate but different variation across repeated audits. Ask whether both are plausible given expected changes in staffing, case mix and policy. Updating later combines these assumptions with observed counts; it cannot make a poor prior choice disappear from a tiny audit.

Preserve the pre-data record

Save the outcome definition, audit eligibility rule, prior parameters, predictive checks and reasons for accepting or revising the prior. If the current data influenced the prior, disclose that dependence and avoid counting the same evidence twice. For a new queue with no credible prior period, use a weak, defensible prior and present sensitivity to alternatives.

Implementation

def prior_probability_of_zero_events(alpha, beta, audit_cases):
    if alpha <= 0 or beta <= 0 or audit_cases < 0:
        raise ValueError("positive beta shapes and nonnegative case count required")
    probability = 1.0
    for completed_cases in range(audit_cases):
        probability *= (beta + completed_cases) / (alpha + beta + completed_cases)
    return probability

zero_event_probability = prior_probability_of_zero_events(3, 27, 47)
assert round(zero_event_probability, 3) == 0.052

Performance and operating cost

The product uses O(N) time and O(1) space for N audit cases. With a very large N, multiply in log space to reduce underflow. Prior predictive simulation over S audits adds O(SN) draws if every case is generated individually.

Common Mistakes

Do not choose the prior after seeing the current audit and present it as independent.
Do not interpret prior concentration as literal historical case count.
Do not let a rate prior conceal a broken sampling frame.