Confidence intervals: interpret coverage and precision honestly

Last updated: 5 Oct 20265 min read

tutorial

A confidence interval describes the long-run behavior of a construction method under its sampling assumptions.

Interpret the method

A 95% interval procedure should cover a fixed population quantity in about 95% of repeated samples when its assumptions hold. The already-computed interval either contains that quantity or does not; the confidence percentage is not a posterior probability assigned to this one interval. Bootstrap interpretation] supplies a concrete resampling example.

Separate width from validity

More independent observations often narrow an interval. They do not fix selection bias, measurement errors or a misdefined denominator. A very narrow interval around a review rate based only on successfully processed tickets can be precisely wrong for all submissions. Report coverage, unit count and major exclusions beside endpoints.

Use the right resampling unit

If customers or stores create correlated observations, construct uncertainty around those clusters. If the statistic is a ratio, recompute the numerator and denominator in every resample rather than averaging row-level ratios. Cluster bootstrap] makes this distinction explicit.

Check decisions at the boundary

Suppose a release requires review rate below 0.14. An estimate of 0.12 with an upper interval endpoint of 0.17 does not establish the release criterion. State whether the rule uses the point estimate or a conservative bound before seeing data. Do not switch the confidence level after observing the result.

Implementation

def interval_release_decision(lower_bound, upper_bound, maximum_rate):
    if not 0 <= lower_bound <= upper_bound <= 1:
        raise ValueError("invalid rate interval")
    if not 0 <= maximum_rate <= 1:
        raise ValueError("invalid rate limit")
    return {"passes_upper_bound_rule": upper_bound < maximum_rate,
            "interval": (lower_bound, upper_bound)}

Performance and operating cost

A parametric interval can be O(N) after sufficient summaries; a bootstrap costs repeated O(N) statistic evaluations. The dominant risk is often invalid sampling assumptions rather than compute.

Common Mistakes

Do not say the fixed parameter has a 95% chance of lying in a frequentist interval.
Do not treat narrow width as proof of unbiased sampling.
Do not choose a bound rule after seeing the endpoints.