Pinball Loss (a.k.a. Quantile Loss)

Pinball loss — also called quantile loss — is the scoring function that makes a model estimate a quantile instead of a mean. For a forecast \(f_{p,t}\) of the \(p\) quantile at time \(t\):

\[Q_{p,t} = \begin{cases} 2(1-p)(f_{p,t} - y_t), & y_t < f_{p,t} \\ 2p\,(y_t - f_{p,t}), & y_t \ge f_{p,t} \end{cases}\]

The name comes from its shape: it is “sometimes called the ‘pinball loss function‘ because a graph of it resembles the trajectory of a ball on a pinball table.” Lower is better — “a low value of \(Q_{p,t}\) indicates a better estimate of the quantile” — and the factor of 2 is a convention: “the multiplier of 2 is often omitted, but including it makes the interpretation a little easier.”

Why the weights are uneven

Read the two branches at \(p = 0.9\). Being too low is penalized at weight \(2p = 1.8\); being too high at \(2(1-p) = 0.2\). A nine-to-one asymmetry, targeting the 0.9 quantile. With a continuous distribution its exceedance probability is 10%; with point masses the quantile condition allows less. For 94% zeros and 6% ones, the 0.9 quantile is zero.

The source gives the continuous-distribution intuition from the comparison side. A quantile forecast means “we expect the observation \(y_t\) to be less than \(f_{p,t}\) with probability \(p\),” so when you compare an outcome with a low quantile such as the 10th percentile, “we need to allow for the fact that it is more likely to be above than below.” A symmetric loss would treat those two directions as equally surprising, and they are not.

One anchor makes the family easy to remember: “when \(p = 0.5\), the quantile score \(Q_{0.5,t}\) is the same as the absolute error.” Median regression is absolute-error regression, and pinball loss generalizes it in both directions.

The mistake it prevents

Scoring a quantile forecast with a symmetric measure rewards the wrong behavior. Evaluate a 0.9 quantile by MAE or RMSE and the best-scoring forecast is not the 0.9 quantile at all — it is the median or the mean, because those minimize those losses. The consequence in practice is a safety-stock or capacity model that gets “improved” until it no longer protects the tail it was built for.

The general principle is the same one that governs point forecasts: “a forecast method that minimises the MAE will lead to forecasts of the median, while minimising the RMSE will lead to forecasts of the mean.” The loss chooses the estimand — so it has to match the quantity you wanted. See forecast error and point forecasts.

Pinball loss does double duty, and it is worth noticing that it is the same function in both roles: as a training objective it is what quantile regression minimizes, and as an evaluation metric it is how a quantile forecast is scored on a test set.

Scoring more than one quantile

Two aggregations build on it, both documented in the same place.

• An interval — the Winkler score, which equals \((Q_{\alpha/2,t} + Q_{1-\alpha/2,t})/\alpha\); it is “the length of the interval plus a penalty if the observation is outside the interval.” See prediction intervals.
• The whole distribution — average “the quantile scores over all values of \(p\)” and you have CRPS.

Which suggests the reporting habit: score the quantile that drives the decision with pinball loss, and if several do, report CRPS as well rather than averaging unrelated numbers. And check calibration separately — a good score can coexist with a level that is exceeded far more often than its label implies. See quantile level.

References

• Hyndman & Athanasopoulos, Forecasting: Principles and Practice (3rd ed.), §5.9 “Evaluating distributional forecast accuracy”
• §5.8 “Evaluating point forecast accuracy”
• §5.5 “Distributional forecasts and prediction intervals”