Murphy’s Decomposition

Murphy’s decomposition separates the binary Brier score into reliability, resolution and uncertainty. This explains whether probability errors come from miscalibration, limited differentiation between cases, or the variability of the outcomes. For the exact sample identity, group observations by distinct forecast value. Let N be the total count, n_m the count in group m, p_m its common forecast, ȳ_m its observed event rate, and ȳ the overall event rate. The identity is:

\[ \text{BS} = \underbrace{\sum_m \tfrac{n_m}{N}(p_m - \bar{y}_m)^2}_{\text{reliability}} - \underbrace{\sum_m \tfrac{n_m}{N}(\bar{y}_m - \bar{y})^2}_{\text{resolution}} + \underbrace{\bar{y}(1 - \bar{y})}_{\text{uncertainty}} \]

The three terms, and their signs

The signs are the part that gets garbled, and getting them wrong inverts the meaning of two of the three terms.

The name “reliability” is the trap. In ordinary speech a reliable forecaster is a good one, and here the reliability term is an error that you want to be zero. Reading it as a quantity to maximize reverses the conclusion of the whole decomposition.

Resolution is defined on outcomes rather than on predictions. A model that issues wildly varying probabilities but whose bins all turn out to have the same event rate has no resolution at all — the spread has to appear in what happened, not in what was said.

Work through the three terms

Consider 100 forecasts: 50 cases receive probability 0.2 and contain 10 positives; the other 50 receive probability 0.8 and contain 40 positives. The overall event rate is 0.5, so uncertainty is 0.5 × 0.5 = 0.25. Each forecast equals its group’s observed rate, giving reliability zero. Resolution is 0.5 × (0.2 − 0.5)² + 0.5 × (0.8 − 0.5)² = 0.09. The Brier score is therefore 0 − 0.09 + 0.25 = 0.16.

If the same two groups were instead assigned 0.1 and 0.9, the outcomes and grouping would be unchanged. Resolution would remain 0.09, but reliability would become 0.5 × (0.1 − 0.2)² + 0.5 × (0.9 − 0.8)² = 0.01. Brier loss would rise to 0.17. The extra extremity adds error here because the forecasts no longer match their observed group rates.

A constant forecast of 0.5 on these data has reliability zero and resolution zero, giving Brier loss 0.25. It is calibrated at that one forecast value but does not distinguish the two groups. This is why calibration alone is not sufficient for useful probability prediction. A constant forecast at another value would add reliability error. The sample base rate is a descriptive benchmark; a deployable baseline must be estimated from information available before the test outcomes are known.

What it is good for

• Diagnosing a bad score. A large reliability term with substantial resolution motivates inspecting the probability mapping. Possible repairs include Platt scaling or isotonic regression on held-out data, but improvement is not guaranteed. Resolution measures outcome separation across forecast groups, not whether higher scores rank positive cases above negative ones. Inspect ordering and the reliability curve before choosing a monotone correction.
• Comparing across datasets, carefully. The uncertainty term changes with prevalence. Removing it does not make different populations equally difficult or directly comparable. Compare models on a common evaluation population where possible and state the baseline when reporting skill.
• Separating two goals. Proper scoring rules reward calibration and sharpness together, and this is what pulls them apart. Sharpness concerns concentration of the predictive distributions; resolution concerns differences in observed rates across forecast groups. They are related in forecast evaluation but are not interchangeable terms.

Its limits

For continuous probabilities, grouping every distinct value can leave only one case per group. The algebraic identity still holds, but its sample terms may be poor estimates of population calibration and resolution. Binning is commonly used to pool evidence. Substituting each bin’s mean probability and observed rate in the three-term expression does not generally reconstruct the original unbinned Brier score exactly; variation and prediction–outcome association within bins create a residual. State whether the reported score is for the original forecasts or a coarsened forecast, and report rather than hide that residual.

The terms are estimated from finite observations. Noise can make the reliability term positive for a calibrated population forecast and inflate apparent resolution in small groups. This is related to the sampling problem in expected calibration error. A small estimated reliability term is evidence to interpret with counts, bin choices and uncertainty, not proof of exact calibration.

The decomposition originates in weather forecasting, where Murphy set it out in 1973, and the vocabulary still carries that history. The underlying question it answers has not changed: when a probabilistic forecast scores badly, is it the forecaster or the weather?

References

• Allan H. Murphy, “A new vector partition of the probability score”, Journal of Applied Meteorology 12(4), 595–600, 1973
• Glenn W. Brier, “Verification of forecasts expressed in terms of probability”, Monthly Weather Review 78(1), 1–3, 1950
• Tilmann Gneiting and Adrian E. Raftery, “Strictly Proper Scoring Rules, Prediction, and Estimation”, Journal of the American Statistical Association 102(477), 359–378, 2007