Probability calibration: test whether risk scores mean what they say

Last updated: 5 Oct 20265 min read

tutorial

A calibrated risk score is consistent with observed outcome frequency over comparable predictions; discrimination and calibration are different properties.

Ask what the number means

If a receipt model assigns about 0.7 risk to many cases, roughly seven in ten of those cases should have the target outcome under a well-calibrated, stable population. A model may rank risky cases correctly while its numeric probabilities are too high. That matters when a threshold, expected-loss calculation or staffing forecast uses the number directly. Decision policy] should state whether it needs calibrated risk or only ordering.

Evaluate on disjoint data

Group held-out predictions into probability bands and compare average score with observed event rate and count. Small bands are noisy; show counts. Assess a proper scoring rule along with the reliability view, while remembering that an aggregate score blends several properties. A calibrator must learn from predictions on data independent of the estimator’s own fit; otherwise training optimism can make probabilities look more certain than they are.

Avoid circular repair

A calibration method can improve probabilities on the calibration partition yet fail after a source or population change. Hold out a separate final period. Examine important groups because a single aggregate curve may conceal opposite errors. If labels mature slowly, exclude or mark pending records instead of treating every unlabeled case as negative. Time-aware evaluation] handles that boundary.

Watch the deployed curve

Monitor score distribution, feature completeness, outcome rate and calibration once labels arrive. A shifted class prevalence can change the interpretation of scores without any code change. Recalibration is a versioned model change; repeat the same evaluation gate and review threshold consequences before release.

Implementation

from sklearn.calibration import calibration_curve
from sklearn.metrics import brier_score_loss

observed, predicted = calibration_curve(
    holdout_labels, holdout_probabilities,
    n_bins=8, strategy="quantile")
brier = brier_score_loss(holdout_labels, holdout_probabilities)
for mean_score, event_rate in zip(predicted, observed):
    print(round(mean_score, 3), round(event_rate, 3))
print("brier", round(brier, 4))

Performance and operating cost

Calibration evaluation is O(N) after scoring and uses O(B) summary space for B bins. Fitting a calibrator adds training data requirements and another versioned artifact; sparse bins can create unstable claims.

Common Mistakes

Do not equate a high ranking score with reliable probabilities.
Do not fit calibration on the same predictions used to train the classifier.
Do not read a small bin as a precise population rate.