Strictly Proper Scoring Rules

1. Definition

• A scoring rule is a function that assigns a numerical score to a forecast, based on the forecast and the observed outcome.
• A scoring rule is proper if the best (lowest expected) score is achieved by reporting the true probability distribution.
• It is strictly proper if the true distribution is the unique minimizer.

In simple words:

• A strictly proper scoring rule encourages honest forecasting.
• If you try to “cheat” by reporting a different probability distribution than your true belief, your expected score will be worse.

---

2. Why They Matter

• In probabilistic forecasts, we don’t just care about accuracy but also about how well the forecast reflects uncertainty.
• Strictly proper scoring rules ensure forecasters are incentivized to provide their true probabilities rather than manipulating outputs.

---

3. Common Strictly Proper Scoring Rules

Logarithmic Score (Log Loss / Negative Log-Likelihood)

$S(p, y) = -\log p(y)$

• Penalizes overconfident wrong forecasts very heavily.
• Widely used in ML classification and probabilistic forecasting.

---

Brier Score

$S(p, y) = \sum_{k=1}^K (p_k – o_k)^2$

• $p_k$​ = predicted probability of class $k$,
• $o_k = 1$ if outcome = class $k$, else 0.
• Works for binary and multiclass events.

---

Continuous Ranked Probability Score (CRPS)

For continuous variables with forecast CDF $F$ and observed outcome $y$:

$CRPS(F,y) = \int_{-\infty}^\infty \big(F(x) – \mathbb{1}\{y \leq x\}\big)^2 dx$

• Generalizes Brier score to continuous outcomes.
• Used a lot in time series forecasting competitions (e.g., M5).

---

4. Example

Suppose you predict the chance of rain tomorrow:

• Forecast A: 90% rain, 10% no rain.
• Forecast B: 60% rain, 40% no rain.
• Actual outcome = “Rain.”
• Log score: Forecast A gets higher reward (because it was more confident and correct).
• If it hadn’t rained, Forecast A would be severely punished, more than Forecast B.

This property encourages well-calibrated probabilities.

---

5. Contrast with Improper Scoring Rules

• Improper scoring rules can be “gamed.”
• Example: Accuracy at threshold 0.5 is not strictly proper → forecasters may distort probabilities to maximize accuracy.
• Proper scoring rules → best to tell the truth.

---

6. Applications

• Weather forecasts: evaluate probabilistic predictions of rain, temperature.
• Finance: risk forecasts (VaR, distributions of returns).
• Machine Learning: probabilistic classifiers (log loss, Brier).
• Forecasting competitions (M5, M6): CRPS and pinball loss are standard.

---

Summary:
A strictly proper scoring rule is a metric for evaluating probabilistic forecasts that uniquely incentivizes truthful reporting of probabilities. Examples include Log Score, Brier Score, and CRPS. They are central to evaluating probability forecasts and ensuring models are calibrated and honest about uncertainty.