Brier Score

The Brier score is the mean squared error of probability forecasts. For binary outcomes, with \(\hat{p}_i\) the predicted probability and \(y_i\) the observed 0 or 1,

\[ \text{BS} = \frac{1}{N}\sum_{i=1}^{N} (\hat{p}_i - y_i)^2 \]

Lower is better. Predicting 0.8 when the event occurs costs \((0.8 - 1)^2 = 0.04\); predicting 0.8 when it does not costs \((0.8 - 0)^2 = 0.64\). Squaring is what makes confident mistakes expensive, and it is why the score notices things accuracy cannot.

Why squared error and not something else

The Brier score is a strictly proper scoring rule, which is a specific and useful guarantee: a forecaster minimizes their expected score by reporting exactly what they believe. There is no probability other than your honest one that scores better on average. A rule without that property can be gamed — a metric that merely rewards being on the right side of 0.5, for instance, rewards reporting 0 or 1 regardless of belief.

That is the reason to prefer it to a binned summary such as expected calibration error when the question is which of two models to ship. It has no binning parameter and no bias from finite bins. It is not, however, immune to the base-rate forecast: a model can score worse than the constant \(\bar{y}\), as the overconfident forecaster below does.

What it decomposes into

Murphy showed that the score splits into three interpretable parts. The identity is exact when predictions are grouped by distinct forecast value; grouping them into bins instead makes it an approximation, which is what the code below does. With \(\bar{y}\) the overall event rate:

\[ \text{BS} = \underbrace{\sum_m \tfrac{n_m}{N}(\bar{p}_m - \bar{y}_m)^2}_{\text{reliability}} - \underbrace{\sum_m \tfrac{n_m}{N}(\bar{y}_m - \bar{y})^2}_{\text{resolution}} + \underbrace{\bar{y}(1 - \bar{y})}_{\text{uncertainty}} \]

The signs are the content. Reliability is miscalibration and is added, so lower is better and zero is perfect. Resolution is how far the bins’ outcome rates spread away from the base rate — how much the model distinguishes cases — and is subtracted, so more is better. Uncertainty is the variance of the outcome itself, fixed by the data and identical for every model on it.

import numpy as np

def brier(y, p):
    return float(np.mean((p - y) ** 2))

def murphy_decomposition(y, p, bins=10):
    """Brier = reliability - resolution + uncertainty, with predictions grouped into bins."""
    edges = np.linspace(0, 1, bins + 1)
    base_rate = y.mean()
    reliability = resolution = 0.0
    for i in range(bins):
        in_bin = (p > edges[i]) & (p <= edges[i + 1]) if i else (p >= edges[0]) & (p <= edges[1])
        k = in_bin.sum()
        if k:
            observed, predicted = y[in_bin].mean(), p[in_bin].mean()
            reliability += k / len(y) * (predicted - observed) ** 2
            resolution += k / len(y) * (observed - base_rate) ** 2
    uncertainty = base_rate * (1 - base_rate)
    return reliability, resolution, uncertainty

rng = np.random.default_rng(6)
n = 20000
p_true = rng.beta(2, 5, n)
y = (rng.uniform(size=n) < p_true).astype(int)

honest = p_true
overconfident = np.clip(p_true * 1.6, 0, 1)
constant = np.full(n, y.mean())

for name, p in (("honest", honest), ("overconfident", overconfident), ("base rate", constant)):
    rel, res, unc = murphy_decomposition(y, p)
    print(f"{name:14} {brier(y, p):.4f} {rel:.4f} {res:.4f} {unc:.4f} {rel - res + unc:.4f}")
# honest 0.1790 0.0001 0.0241 0.2036 0.1797
# overconfident 0.2172 0.0381 0.0242 0.2036 0.2175
# base rate 0.2036 0.0000 0.0000 0.2036 0.2036

The middle row is the one worth sitting with. The transformation does not reverse the order of cases, but clipping creates ties among the highest probabilities. Its binned resolution is similar at about 0.024, and it scores worse than predicting the base rate every time, because overconfidence added 0.038 of reliability error and there was only 0.024 of resolution to pay for it. Discrimination and calibration are separate properties, and a model can have all of the first and still be worse than useless.

The bottom row also gives the reference point the raw score needs. A constant forecast of the base rate has no reliability error and no resolution, so it scores exactly the uncertainty term, \(\bar{y}(1 - \bar{y})\). Any model that does not beat that number has contributed nothing. The decomposition is computed from bins and is therefore approximate — the three parts here sum to 0.1797 against a measured 0.1790 — so it is a diagnostic rather than an identity to reconcile.

The raw number does not travel

Because uncertainty is part of the score and depends only on the base rate, Brier scores from different datasets are not comparable. A fraud problem with a 1 percent base rate has an uncertainty term of 0.0099, which is what the base-rate forecast itself scores. That is the number to compare against — not a ceiling, since a model is free to do worse (predicting 1 everywhere scores 0.99). A Brier score of 0.008 might be excellent or might be barely better than a constant. A balanced problem has an uncertainty term of 0.25, and 0.2 there is a real achievement.

The fix is to report the score against the constant-base-rate baseline — the skill score, one minus the ratio — or to report the decomposition. Either makes the comparison meaningful; the raw value on its own does not.

Where it is used and where it is not

• Weather forecasting, where the score originated with Brier in 1950 and where a stated 70 percent chance of rain is expected to verify about 70 percent of the time.
• Clinical risk models, where a stated risk is acted on directly and a model that ranks well but reports the wrong levels leads to the wrong thresholds.
• Fraud and credit scoring, with the base-rate caveat above; here the decomposition matters more than the total.

Where it is less useful is when only the ordering matters — a fixed number of cases will be reviewed and nothing depends on the stated probabilities. Then a rank-based measure answers the question, and the Brier score penalizes a miscalibration that has no consequence. The distinction, and when calibration is worth fixing, is under model calibration; the shape of the miscalibration shows up in a reliability curve, and the repair in Platt scaling.

References

• Glenn W. Brier, “Verification of forecasts expressed in terms of probability”, Monthly Weather Review 78(1), 1–3, 1950
• Allan H. Murphy, “A new vector partition of the probability score”, Journal of Applied Meteorology 12(4), 595–600, 1973
• Tilmann Gneiting and Adrian E. Raftery, “Strictly Proper Scoring Rules, Prediction, and Estimation”, Journal of the American Statistical Association 102(477), 359–378, 2007
• scikit-learn User Guide, Probability calibration