Monte Carlo estimands and reproducible draws

Last updated: 5 Oct 20265 min read

tutorial

A Monte Carlo estimate averages outcomes from a specified stochastic model; the target quantity must be defined before generating random draws.

Name the operational quantity

A support lead has capacity for 14 urgent reviews on a shift with 61 eligible cases. Suppose each case independently requires urgent review with modeled probability 0.19. The immediate question is the model-based chance that urgent demand exceeds 14, not the average number of urgent cases and not an observed historical breach rate. One simulated shift draws 61 urgent flags and records a breach indicator. Averaging that indicator over independent simulated shifts estimates the modeled breach probability. The decision unit is one shift.

Separate inputs from outcomes

The case count, urgent-share model, capacity and breach rule are inputs. The number of urgent cases and whether demand exceeds capacity are outputs. A change from more than 14 to at least 14 changes the event; document the boundary in the metric contract. The independence assumption is also an input, not a fact created by running code. Shared campaign days or weather shocks can make urgent cases arrive together. Joint-input modeling handles that case.

Keep a reproducible random stream

Use a local random generator initialized with a recorded seed. Identical code, configuration and runtime behavior then replay the same pseudo-random sequence in this environment. A seed helps investigate a calculation; it does not make an uncertain assumption true, nor does one seed create independent evidence. Store the code version, input snapshot, run count and seed together. An analysis snapshot should let a reviewer recover the exact model release.

Distinguish model risk from numerical noise

If the urgent probability is actually 0.27, adding simulations around 0.19 only estimates the wrong model more precisely. Examine historical case counts and stratify by shift type before fitting an input distribution. Hold back recent shifts for validation and compare predicted breach frequencies with observed ones. A simulator should fail visibly when a parameter falls outside its domain rather than silently clipping it to a comfortable value.

Use the output for a decision

A modeled breach probability has value only beside an action threshold and consequence. One manager may buy flex coverage if breach chance exceeds 0.12; another may require expected backlog cost to exceed a staffing price. Preserve the probability estimate, input assumptions and candidate policy separately. Simulation error tells how much numerical precision the chosen run count bought, while the project carries the result into a policy comparison.

Implementation

from random import Random

def urgent_review_breach_probability(cases_per_shift, urgent_probability,
                                     review_capacity, replications, seed):
    if cases_per_shift < 0 or not 0 <= urgent_probability <= 1:
        raise ValueError("invalid demand model")
    if review_capacity < 0 or replications <= 0:
        raise ValueError("invalid capacity or run count")
    draw_stream = Random(seed)
    breaches = 0
    for _ in range(replications):
        urgent_cases = sum(draw_stream.random() < urgent_probability
                           for _ in range(cases_per_shift))
        breaches += urgent_cases > review_capacity
    return breaches / replications

first_run = urgent_review_breach_probability(61, 0.19, 14, 24000, 4817)
replayed = urgent_review_breach_probability(61, 0.19, 14, 24000, 4817)
assert first_run == replayed
assert 0 < first_run < 1

Performance and operating cost

The direct model draws C case flags for each of R shifts: O(R × C) time and O(1) additional space. This is suitable for a transparent baseline; a binomial sampler can reduce work at scale. Faster sampling does not compensate for a wrong input distribution.

Common Mistakes

Do not call a modeled breach probability an observed historical rate.
Do not change a strict greater-than rule to greater-than-or-equal during reporting.
Do not treat a fixed seed as evidence that the input model is valid.