Maximum Calibration Error (MCE)

Maximum calibration error reports the worst bin rather than the average one. Predictions are grouped into bins exactly as for expected calibration error, each bin’s gap between what was predicted and what happened is computed, and the largest of those gaps is the score:

\[ \text{MCE} = \max_{m}\ \big| \text{obs}(B_m) - \text{pred}(B_m) \big| \]

The motivation is sound. ECE weights each bin by how many predictions it holds, so a region that is badly wrong but rarely visited contributes almost nothing — and in a fraud or screening model that sparse high-probability region is exactly where decisions get made. MCE ignores the weighting and asks how bad the worst place is.

As with ECE, two versions share the name: one comparing the predicted positive-class probability with the observed rate, and one comparing top-label confidence with accuracy. They answer different questions and are not interchangeable.

Why it is noisier than ECE

Taking a maximum instead of a weighted average has a cost that is easy to underestimate. Each observed bin gap contains sampling variation. ECE averages absolute gaps, so their signs do not cancel; weighting by bin frequency limits the contribution of sparse bins. MCE instead selects the largest observed gap. Comparing many bins can select a large sampling fluctuation even when the underlying probabilities are calibrated.

import numpy as np

def perfectly_calibrated(n, seed):
    rng = np.random.default_rng(seed)
    p = rng.uniform(0, 1, n)
    return (rng.uniform(size=n) < p).astype(int), p

def bin_gaps(y, p, bins):
    """Per-bin |observed rate - mean prediction|, with the bin's share of the sample."""
    edges = np.linspace(0, 1, bins + 1)
    out = []
    for i, (lo, hi) in enumerate(zip(edges[:-1], edges[1:])):
        m = (p >= lo) & (p <= hi) if i == 0 else (p > lo) & (p <= hi)
        if m.sum():
            out.append((m.sum() / len(y), abs(y[m].mean() - p[m].mean()), m.sum()))
    return out

def ece(y, p, bins):
    return sum(w * g for w, g, _ in bin_gaps(y, p, bins))

def mce(y, p, bins):
    return max((g for _, g, _ in bin_gaps(y, p, bins)), default=float("nan"))

def smallest_bin(y, p, bins):
    return min((n for _, _, n in bin_gaps(y, p, bins)), default=0)

# A model that is exactly right, scored 40 times at each binning.
for bins in (5, 10, 20, 50):
    runs = [perfectly_calibrated(2000, seed=s) for s in range(40)]
    e = np.mean([ece(y, p, bins) for y, p in runs])
    m = np.mean([mce(y, p, bins) for y, p in runs])
    k = np.mean([smallest_bin(y, p, bins) for y, p in runs])
    print(f"{bins:>2} {e:.4f} {m:.4f} {m / e:.1f} {k:.0f}")
#  5 0.0165 0.0338 2.1 376
# 10 0.0226 0.0560 2.5 179
# 20 0.0313 0.0867 2.8 82
# 50 0.0495 0.1717 3.5 27

Every number in that table comes from a model whose probabilities are exactly right, averaged over forty replicates of two thousand predictions. At fifty bins it reports a maximum calibration error of 0.17 — a figure that, read at face value, says some probability bucket is off by seventeen percentage points. Nothing is off by anything; the smallest bins hold about twenty-seven observations each, and a proportion from twenty-seven observations wanders by that much routinely.

The ratio column is the part to carry away. ECE’s upward bias grows roughly with the square root of the bin count; MCE’s grows faster, because it is an extreme value drawn from more and more candidates. So the two metrics diverge exactly as the binning gets finer, which is when a practitioner is most likely to be looking for a localized problem.

Reading it without being misled

• Cover the range exactly once. Half-open bins of the form \(p > \text{lo}\), \(p \le \text{hi}\) drop \(p = 0\) entirely; with a model that predicts exact zeros the maximum is taken over the wrong set, and over an empty set it fails outright. Close the first bin on the left and decide what an empty input returns.
• Report the bin that produced it. A maximum with no location is not actionable, and knowing it came from the 0.9 to 1.0 bin rather than the 0.4 to 0.5 bin changes what to do about it entirely.
• Report that bin’s size. A gap of 0.2 from four hundred observations is a finding; from twenty it may or may not be, and the count alone does not decide. Where the bin’s predictions are near 0.5 a gap that size is ordinary; where they are near 0.001, seeing four positives in twenty has probability about \(4.8 \times 10^{-9}\) under perfect calibration, so the same gap is decisive. The stated probabilities, not just the count, set the scale.
• Compare against a perfect model. Simulate outcomes from the model’s own predicted probabilities — not from a uniform draw with the same sample size — and recompute the maximum the same way, including the step of taking a maximum over many bins, which inflates it on its own. That gives the number “no miscalibration” actually looks like for this model.
• Use equal-count bins. Sparse bins are where the maximum comes from, and balancing bin sizes removes the worst of that — though it does not remove the bias, as the measurements under adaptive ECE show.

When it earns its place

MCE and ECE answer different questions and neither substitutes for the other. ECE asks how far off the probabilities are on average across the cases the model actually sees. MCE asks whether there is any region where the model is dangerously wrong, which is the right question when the cost of a mistake is concentrated — a clinical threshold, an automated rejection, a trade size set from a stated probability.

For diagnosis, though, a reliability curve with bin counts shown does everything MCE does and more: it shows where the worst bin is, how large it is, and how much evidence stands behind it, without compressing all three into one fragile number. MCE is best understood as a summary of that plot for people who will not look at it — and like any such summary, it should arrive with the plot rather than instead of it.

Neither metric is a good optimization target. Both can be driven to zero by predicting the base rate for every case, which destroys every bit of information the model had; that is what the proper scoring rules under Brier score exist to prevent.

References

• Chuan Guo, Geoff Pleiss, Yu Sun and Kilian Q. Weinberger, “On Calibration of Modern Neural Networks”, ICML 2017, arXiv:1706.04599
• scikit-learn User Guide, Probability calibration