Conditional quantiles: optimize tail predictions with pinball loss

Last updated: 7 Oct 20265 min read

tutorial

A conditional quantile predicts a percentile of outcomes at given feature values, which differs from the mean and from a pooled population percentile.

State the quantity for the decision

A dispatch team needs a planned time that covers roughly nine of ten comparable shipments, conditional on route, service tier, and booking day. A global ninetieth percentile ignores those fields; a conditional mean answers another question. Fit a model for the chosen quantile level, then evaluate it on later shipments whose outcomes were not used to fit the model. The distribution lesson addresses unconditional summaries and the treatment of extreme observations.

Use an asymmetric loss

Pinball loss weights an underprediction by the quantile level and an overprediction by one minus that level. At a high quantile, missing a slow shipment is costly in the objective, so its minimizer estimates a high conditional percentile under appropriate model capacity and data. The code below scores supplied predictions; it does not train a quantile model. An intercept-only minimizer targets a marginal quantile, while a feature model targets conditional behavior. Prediction intervals answer a related but distinct two-bound question.

Separate fit from calibration

A small average pinball loss does not prove that each route tier has the desired empirical coverage. Count how often actual dispatch time is at or below the predicted threshold overall and within meaningful groups on a time-separated holdout. Too few examples in a rare route make its coverage noisy; report numerator and denominator. Shipment cancellations and late outcome logging can distort the holdout just as they distort training data.

Know what a percentile promise means

A predicted ninetieth percentile is not a guarantee for the next specific shipment. The conditional model can be misspecified, and traffic patterns can change. If the service promise has a penalty, compare alternative quantile levels against the actual loss and monitoring threshold. The coverage lesson checks ordering and group performance when several quantiles are estimated separately; the project turns those checks into a release gate.

Implementation

def mean_pinball(actual_minutes, predicted_minutes, quantile):
    if not 0 < quantile < 1 or not actual_minutes or        len(actual_minutes) != len(predicted_minutes):
        raise ValueError("aligned values and a quantile in (0, 1) required")
    return sum(quantile * max(actual - predicted, 0) +
               (1 - quantile) * max(predicted - actual, 0)
               for actual, predicted in zip(actual_minutes, predicted_minutes)) /            len(actual_minutes)

assert round(mean_pinball([31, 52], [38, 44], 0.9), 2) == 3.95

Performance and operating cost

Scoring n forecasts is O(n) time and O(1) extra space. Fitting a conditional quantile model is usually more expensive and depends on the chosen model family. A global sorted percentile is cheaper but misses route-specific variation and can under-cover slow groups.

Common Mistakes

Calling a conditional quantile a mean prediction.
Reporting training-set coverage as the service guarantee.
Dropping delayed or cancelled shipments from holdout without a defined rule.
Using one global coverage figure to hide a failing route tier.