1. Definition

  • Probabilistic scoring = evaluating probabilistic forecasts (forecasts that give distributions or probabilities) using scoring rules.
  • A scoring rule is a function $S(f,y)$ that measures how well a forecast distribution $f$ matches the observed outcome $y$.

Unlike accuracy (for point forecasts), probabilistic scoring checks:

  • Are probabilities calibrated? (Do 70% forecasts rain ~70% of the time?)
  • Are forecasts sharp? (Do they give precise, informative distributions?)

2. Proper and Strictly Proper Scoring Rules

  • A scoring rule is proper if the best expected score comes from reporting the true distribution.
  • It is strictly proper if the true distribution is the unique optimum.
    This encourages honest, well-calibrated probability forecasts.

3. Common Probabilistic Scoring Rules

Logarithmic Score (Log Loss, Negative Log-Likelihood)

$S(p,y) = -\log p(y)$

  • Penalizes overconfident wrong forecasts heavily.
  • Widely used in ML classification.

Brier Score (for discrete outcomes)

$S(p,y) = \sum_{k=1}^K (p_k – o_k)^2$

  • $p_k$​ = forecast probability for class $k$.
  • $o_k = 1$ if observed outcome = class $k$, else 0.
  • Measures squared error between forecast probabilities and outcome.

CRPS (Continuous Ranked Probability Score) (for continuous outcomes)

$CRPS(F,y) = \int_{-\infty}^\infty \big(F(x) – \mathbf{1}\{y \leq x\}\big)^2 dx$

  • Compares predicted CDF $F(x)$ to the “step function” defined by the observed value.
  • Like “MAE for distributions.”

Pinball Loss (for quantiles)

$L_\alpha(y, \hat{q}_\alpha) = \begin{cases} \alpha (y – \hat{q}_\alpha), & y \geq \hat{q}_\alpha \\[6pt] (1-\alpha)(\hat{q}_\alpha – y), & y < \hat{q}_\alpha \end{cases}$

  • Used for quantile forecasts (e.g., 10th, 50th, 90th percentiles).

4. Why Probabilistic Scoring Matters

  • Point forecast error metrics (MAE, RMSE) ignore uncertainty.
  • Probabilistic scoring rewards models that produce both accurate and well-calibrated probability distributions.
  • Essential in:
    • Weather: “70% chance of rain” must be reliable.
    • Finance: Value-at-Risk forecasts.
    • Energy & demand forecasting: safety margins depend on probabilistic forecasts.
    • Forecasting competitions (M4, M5): CRPS and pinball loss are standard metrics.

5. Summary

  • Probabilistic scoring = evaluation of probabilistic forecasts using proper scoring rules.
  • Key metrics: Log Score, Brier Score, CRPS, Pinball Loss.
  • They ensure forecasts are not just sharp but also calibrated and truthful.
  • “Strictly proper” → incentivizes reporting the true distribution.