Probabilistic Scoring

Probabilistic scoring is the evaluation of forecasts that report distributions or probabilities rather than single values. A scoring rule is a function \(S(f, y)\) measuring how well a forecast \(f\) matches the observed outcome \(y\), and unlike accuracy it has two jobs: checking whether the probabilities are calibrated — do 70% forecasts come true about 70% of the time — and whether they are sharp, meaning informative rather than hedged.

The property that makes a rule usable is propriety. A rule is proper if the best expected score comes from reporting your true distribution, and strictly proper if that is the only optimum. The word does a lot of work, and it is worth seeing what it excludes.

What propriety actually rules out

import numpy as np

p = 0.7
"""The event happens with probability 0.7. A forecaster who knows that
   reports q, and we ask which q each rule rewards."""

def expected(rule, q):
    return p * rule(q, 1) + (1 - p) * rule(q, 0)

log_score = lambda q, y: -np.log(q if y else 1 - q)
brier = lambda q, y: (q - y) ** 2
linear = lambda q, y: abs(q - y)
accuracy = lambda q, y: float((q >= 0.5) != y)

print(f"  true probability {p}. Expected penalty for reporting q:")
print(f"{'reported q':>12}{'log score':>12}{'Brier':>10}{'|q - outcome|':>15}{'error rate':>12}")
for q in (0.5, 0.6, 0.7, 0.8, 0.9, 0.99):
    print(f"{q:>12}{expected(log_score, q):>12.6f}{expected(brier, q):>10.6f}"
          f"{expected(linear, q):>15.6f}{expected(accuracy, q):>12.4f}")

grid = np.linspace(0.001, 0.999, 999_999)
for name, rule in (("log score", log_score), ("Brier", brier), ("|q - outcome|", linear)):
    e = np.array([p * rule(grid, 1) + (1 - p) * rule(grid, 0)]).ravel()
    print(f"  {name:>14} is minimized at q = {grid[e.argmin()]:.4f}")

print(f"\n  what a proper rule buys, over 100,000 forecasts at p = {p}:")
rng = np.random.default_rng(0)
truth = (rng.random(100_000) < p).astype(float)
for label, q in (("honest 0.70", 0.70), ("shaded to 0.80", 0.80), ("claimed 0.99", 0.99)):
    print(f"{label:>18}  log {(-np.log(np.where(truth == 1, q, 1 - q))).mean():>9.6f}"
          f"   Brier {((q - truth) ** 2).mean():>9.6f}"
          f"   correct {np.mean((q >= 0.5) == truth):>7.4f}")

for q in (0.1, 0.01, 0.001, 0.0):
    with np.errstate(divide="ignore"):
        print(f"    reporting {q:>6} when the event happens costs"
              f" {-np.log(q) if q > 0 else float('inf'):>10.4f}")
#   true probability 0.7. Expected penalty for reporting q:
#   reported q   log score     Brier  |q - outcome|  error rate
#          0.5    0.693147  0.250000       0.500000      0.3000
#          0.6    0.632465  0.220000       0.460000      0.3000
#          0.7    0.610864  0.210000       0.420000      0.3000
#          0.8    0.639032  0.220000       0.380000      0.3000
#          0.9    0.764528  0.250000       0.340000      0.3000
#         0.99    1.388586  0.294100       0.304000      0.3000
#        log score is minimized at q = 0.7000
#            Brier is minimized at q = 0.7000
#    |q - outcome| is minimized at q = 0.9990
#
#   what a proper rule buys, over 100,000 forecasts at p = 0.7:
#        honest 0.70  log  0.610178   Brier  0.209676   correct  0.7008
#     shaded to 0.80  log  0.637909   Brier  0.219514   correct  0.7008
#       claimed 0.99  log  1.384864   Brier  0.293306   correct  0.7008
#     reporting    0.1 when the event happens costs     2.3026
#     reporting   0.01 when the event happens costs     4.6052
#     reporting  0.001 when the event happens costs     6.9078
#     reporting    0.0 when the event happens costs        inf

The rules in use

Four cover most of the ground, and each is the natural rule for a different shape of forecast.

• Logarithmic score, \(S(p, y) = -\log p(y)\) — the negative log-likelihood, and the standard loss for classifiers. It punishes confident errors without bound, as the last rows above show.
• Brier score, \(S(p, y) = \sum_{k=1}^{K}(p_k - o_k)^2\) for discrete outcomes, where \(o_k\) is 1 for the observed class and 0 otherwise — squared error between the forecast probabilities and the outcome.
• CRPS, \(\int_{-\infty}^{\infty}\big(F(x) - \mathbf{1}\{y \leq x\}\big)^2 dx\) for continuous outcomes — the distance between the predicted CDF and the step function at the observation, and the closest thing to a mean absolute error for distributions.
• Pinball loss for quantile forecasts: \(\alpha(y - \hat{q}_\alpha)\) when \(y \geq \hat{q}_\alpha\) and \((1-\alpha)(\hat{q}_\alpha - y)\) otherwise.

All four are proper. Choosing among them is therefore not a question of which is correct but of how you want confident errors treated, and that is a decision about the application rather than about statistics. The log score is the strict end: one event you called impossible and then observed makes the average infinite, so a single mislabeled row can end a model comparison. The binary single-probability Brier score used in the code caps it at 1.0; the sum over all classes in the definition above has maximum 2.0. In weather and in medicine the unbounded penalty is usually wanted; in a leaderboard over noisy labels it often is not.

Pinball loss is the odd one out in a useful way: it scores a single cut point rather than a whole distribution, so a set of quantiles is scored by summing it over them. That makes it the standard metric where the decision is a threshold — a stock level, a capacity limit — rather than a full distribution.

Why it is not optional

Point-forecast error metrics such as MAE and RMSE take a single number and compare it to what happened. They cannot express uncertainty, because there is no uncertainty in their input. Once a model outputs a probabilistic forecast, scoring it with a point metric throws away the part that was expensive to produce.

The middle table above is the practical argument. Three forecasters who say 0.70, 0.80 and 0.99 are all correct on exactly the same days, so any threshold-based metric ranks them equal. The scores separate them immediately, and they do so in the direction that matters: the one telling the truth wins.

That is why these rules are the reporting standard where probabilities carry consequences — a 70% chance of rain has to be reliable, a value-at-risk forecast is a risk forecast someone sets capital against, and safety margins in energy and demand planning are read off the tails. Competition metrics differ by task: M4 used OWA (combining relative sMAPE and MASE) for point forecasts and MSIS for prediction intervals; M5 Uncertainty used weighted scaled pinball loss. Proper scoring rules encourage honest probability reports in expectation, not guaranteed wins on each realized sample.

References

• NumPy documentation, numpy.random.Generator. numpy.org
• University of Nicosia, M4 Competition evaluation. Source