Label-prior shift and odds adjustment

Last updated: 5 Oct 20265 min read

tutorial

A prior-shift odds adjustment changes a calibrated binary posterior when the class prevalence changes but the feature distribution within each class remains stable.

Name the strong assumption

Suppose the missed-handoff share rises from an older cohort to a recent one. Prior shift assumes that, conditional on whether a handoff is missed, the distribution of intake features is unchanged. Only the class prior changes. A new scanner, route policy or crew allocation can violate that assumption. The code shows the odds conversion under the assumption; it cannot diagnose whether the assumption holds. Feature monitoring gives a first warning, not proof.

Use independent prevalence evidence

The old prior should come from the population the original calibrated model represented. Estimate the new prior from a properly matured, representative labeled cohort or a validated quantification method. An actively sampled review queue overrepresents uncertain cases and cannot provide it directly. Nor can the same-day fraction of resolved shipments when slow failures remain pending. Active sampling and label maturity explain those traps.

Adjust odds, not raw probability by addition

Convert the old posterior to odds, multiply by the ratio of new prior odds to old prior odds, then convert back to probability. This follows from the unchanged class-conditional feature likelihoods. The code checks that a baseline posterior equal to the old prior maps to the new prior. Directly adding the prevalence difference can leave the interval from zero to one and is not the same operation.

Recheck calibration and decisions

Even when the algebra is correct, the old probabilities may be miscalibrated and the new prevalence estimate noisy. Evaluate adjusted predictions on a later independent labeled cohort. Review log loss, calibration and the number of dispatch interventions. A changed probability may move cases across the cost threshold, so the resulting policy needs a new capacity check. Calibration and threshold costing remain separate tests.

Prefer retraining when the mechanism changed

If the scanner schema, carrier mix or relation between backlog and outcome changed, an odds multiplier cannot repair the model. Compare the adjusted candidate with a recent-data refit under the same evaluation boundary. Record which assumption justified the adjustment and when it expires. The project asks for a hold decision when that evidence is absent.

Implementation

def adjust_for_prior_shift(old_probability, old_prior, new_prior):
    if not all(0 < value < 1 for value in
               (old_probability, old_prior, new_prior)):
        raise ValueError("probabilities and priors must be inside (0, 1)")
    old_odds = old_probability / (1 - old_probability)
    prior_odds_ratio = (new_prior / (1 - new_prior)) / (old_prior / (1 - old_prior))
    adjusted_odds = old_odds * prior_odds_ratio
    return adjusted_odds / (1 + adjusted_odds)

old_prior = 0.18
new_prior = 0.31
scores = [0.06, 0.18, 0.42, 0.73]
adjusted = [adjust_for_prior_shift(score, old_prior, new_prior)
            for score in scores]
assert abs(adjust_for_prior_shift(old_prior, old_prior, new_prior) - new_prior) < 1e-12
assert all(new > old for old, new in zip(scores, adjusted))
assert all(0 < score < 1 for score in adjusted)

Performance and operating cost

The adjustment is O(N) time for N scored cases and O(1) working state if streamed. Estimating the new prior, checking class-conditional stability and validating the policy require independent labels and far more work than the conversion itself.

Common Mistakes

Do not infer the new prior from an actively sampled or immature-label cohort.
Do not apply an odds correction when the class-conditional feature process changed.
Do not treat the adjusted output as calibrated without a later labeled check.