Probabilistic Scoring

1. Definition

• Probabilistic scoring = evaluating probabilistic forecasts (forecasts that give distributions or probabilities) using scoring rules.
• A scoring rule is a function $S(f,y)$ that measures how well a forecast distribution $f$ matches the observed outcome $y$.

Unlike accuracy (for point forecasts), probabilistic scoring checks:

• Are probabilities calibrated? (Do 70% forecasts rain ~70% of the time?)
• Are forecasts sharp? (Do they give precise, informative distributions?)

---

2. Proper and Strictly Proper Scoring Rules

• A scoring rule is proper if the best expected score comes from reporting the true distribution.
• It is strictly proper if the true distribution is the unique optimum.
  This encourages honest, well-calibrated probability forecasts.

---

3. Common Probabilistic Scoring Rules

Logarithmic Score (Log Loss, Negative Log-Likelihood)

$S(p,y) = -\log p(y)$

• Penalizes overconfident wrong forecasts heavily.
• Widely used in ML classification.

---

Brier Score (for discrete outcomes)

$S(p,y) = \sum_{k=1}^K (p_k – o_k)^2$

• $p_k$​ = forecast probability for class $k$.
• $o_k = 1$ if observed outcome = class $k$, else 0.
• Measures squared error between forecast probabilities and outcome.

---

CRPS (Continuous Ranked Probability Score) (for continuous outcomes)

$CRPS(F,y) = \int_{-\infty}^\infty \big(F(x) – \mathbf{1}\{y \leq x\}\big)^2 dx$

• Compares predicted CDF $F(x)$ to the “step function” defined by the observed value.
• Like “MAE for distributions.”

---

Pinball Loss (for quantiles)

$L_\alpha(y, \hat{q}_\alpha) = \begin{cases} \alpha (y – \hat{q}_\alpha), & y \geq \hat{q}_\alpha \\[6pt] (1-\alpha)(\hat{q}_\alpha – y), & y < \hat{q}_\alpha \end{cases}$

• Used for quantile forecasts (e.g., 10th, 50th, 90th percentiles).

---

4. Why Probabilistic Scoring Matters

• Point forecast error metrics (MAE, RMSE) ignore uncertainty.
• Probabilistic scoring rewards models that produce both accurate and well-calibrated probability distributions.
• Essential in:
  • Weather: “70% chance of rain” must be reliable.
  • Finance: Value-at-Risk forecasts.
  • Energy & demand forecasting: safety margins depend on probabilistic forecasts.
  • Forecasting competitions (M4, M5): CRPS and pinball loss are standard metrics.

---

5. Summary

• Probabilistic scoring = evaluation of probabilistic forecasts using proper scoring rules.
• Key metrics: Log Score, Brier Score, CRPS, Pinball Loss.
• They ensure forecasts are not just sharp but also calibrated and truthful.
• “Strictly proper” → incentivizes reporting the true distribution.