Strictly Proper Scoring Rules

A scoring rule assigns a number to a probabilistic forecast once the outcome is known. It is proper if reporting your true belief is among the optimal things to do, and strictly proper if it is the only one. That second word is where the guarantee lives.

One convention needs settling first, because the literature and the libraries disagree. Gneiting and Raftery define scoring rules as rewards to be maximized; machine learning implementations define them as losses to be minimized, which is the same thing negated. This entry uses the loss convention throughout — lower is better — because that is what brier_score_loss and log_loss return.

What properness guarantees

Suppose you genuinely believe an event has probability 0.7, and you may report any number you like. A strictly proper rule makes 0.7 the unique report that minimizes your expected score — every alternative is worse, in expectation, by the rule’s own arithmetic. There is no strategy, no hedging, no rounding toward the middle that helps.

That is not a property every metric has, and the difference is easy to compute.

import numpy as np

def expected_score(rule, belief, reported):
    """Average score when the event truly occurs with probability `belief`
    but the forecaster reports `reported` instead."""
    return belief * rule(reported, 1) + (1 - belief) * rule(reported, 0)

brier = lambda p, y: (p - y) ** 2
log_loss = lambda p, y: -np.log(p if y else 1 - p)
# This one never looks at the probability, only at which side of 0.5 it falls.
misclassification = lambda p, y: float((p >= 0.5) != y)

belief = 0.7
reports = np.array([0.3, 0.5, 0.6, 0.7, 0.8, 0.9, 0.99])
for name, rule in (("brier", brier), ("log", log_loss), ("0/1", misclassification)):
    scores = np.array([expected_score(rule, belief, r) for r in reports])
    minimizers = reports[scores <= scores.min() + 1e-12]
    print(f"{name:6} " + " ".join(f"{s:.4f}" for s in scores) + f"  best={[float(m) for m in minimizers]}")
# brier  0.3700 0.2500 0.2200 0.2100 0.2200 0.2500 0.2941  best=[0.7]
# log    0.9498 0.6931 0.6325 0.6109 0.6390 0.7645 1.3886  best=[0.7]
# 0/1    0.7000 0.3000 0.3000 0.3000 0.3000 0.3000 0.3000  best=[0.5, 0.6, 0.7, 0.8, 0.9, 0.99]

Brier and the log score both bottom out at exactly 0.7, and the minimum is unique — they are strictly proper. Misclassification loss is flat across every report from 0.5 upward: six different answers tie for best, so the rule offers no reason to say 0.7 rather than 0.99. It is not strictly proper, and a forecaster optimizing it receives no reward for choosing the truthful 0.7 over an exaggerated 0.99.

This is why accuracy is a poor target when probabilities matter: it gives the forecaster no incentive to report truthful probabilities, and accuracy alone can neither guarantee nor reveal whether the probabilities a model reports are reliable. It is also why a proper rule is the right thing to compare two models on. The optimality is a statement about expectations under the true distribution: no report other than the true probability has a better expected score. It is not a claim that a given model’s output cannot be improved. A model that says 0.99 where the truth is 0.7 has an expected Brier score of 0.2941, and rescaling that output to 0.7 brings it to 0.21 — which is exactly what recalibration does. A proper rule rewards moving toward the truth, from wherever the model currently is.

The rules in common use

One trap in that table is worth pointing at. The multiclass Brier form sums over classes, and for a binary outcome that sum has two terms — \((p - y)^2 + ((1-p) - (1-y))^2 = 2(p - y)^2\). So the multiclass convention gives exactly twice the binary shorthand, and libraries differ on which they return. The difference never changes which model wins; it does make a reported value uninterpretable without the convention.

What a proper rule is actually measuring

A proper score is not a calibration metric. It rewards two things at once: being calibrated, and being sharp — that is, issuing probabilities far from the base rate when the evidence allows it. A forecaster who always reports the base rate is perfectly calibrated and scores badly, because the rule also asks for informativeness.

That combination is the modern framing of probabilistic forecasting: maximize sharpness subject to calibration. Murphy’s decomposition separates the two parts of a Brier score explicitly, which is what makes a proper score diagnostic rather than merely comparative.

Two practical consequences follow. A proper score cannot be compared across datasets, because part of it is the irreducible difficulty of the outcome. And a proper score is the right thing to select a model on, while a decomposition or a reliability curve is the right thing to diagnose one with.

Where it matters

Properness earns its keep wherever the forecaster has an incentive that is not purely truthful — a person whose bonus depends on a metric, a team choosing which model to promote, a vendor reporting its own accuracy. A rule that can be improved by shading probabilities will be, and usually not deliberately: the distortion arrives through threshold tuning, through training on the wrong objective, or through a post-processing step added because it improved the number.

Weather forecasting adopted these rules first and for exactly this reason, and the practice transfers directly. If probabilities will be read, acted on, or combined, score them with a rule that makes honesty optimal — and read the calibration diagnostics separately, under model calibration.

References

• Tilmann Gneiting and Adrian E. Raftery, “Strictly Proper Scoring Rules, Prediction, and Estimation”, Journal of the American Statistical Association 102(477), 359–378, 2007
• Glenn W. Brier, “Verification of forecasts expressed in terms of probability”, Monthly Weather Review 78(1), 1–3, 1950
• scikit-learn User Guide, Metrics and scoring