Adaptive ECE

Adaptive ECE computes expected calibration error over bins holding equal numbers of predictions rather than covering equal widths of the probability range. Sort the predictions, cut them into \(M\) groups of about \(N/M\) each, and the bin boundaries land wherever the predictions happen to be dense.

The motivation is a real problem. A confident classifier puts most of its output near 0 or near 1, so equal-width bins in the middle of the range can hold a handful of predictions or none at all, and a bin holding three observations contributes an essentially random gap. scikit-learn exposes the same choice on its calibration curve through a strategy argument of 'uniform' or 'quantile'.

What it fixes, and what it does not

It is often said that adaptive binning makes the metric less sensitive to the number of bins. Measuring that claim on a model whose probabilities are exactly right shows otherwise.

import numpy as np

def perfectly_calibrated(n, seed, clustered):
    """Predictions that are exactly right. When clustered, most of them sit near 1,
    which is what a confident classifier's output distribution looks like."""
    rng = np.random.default_rng(seed)
    p = rng.beta(8, 1, n) if clustered else rng.uniform(0, 1, n)
    return (rng.uniform(size=n) < p).astype(int), p

def binned_error(y, p, edges):
    total = 0.0
    for i, (lo, hi) in enumerate(zip(edges[:-1], edges[1:])):
        in_bin = (p >= lo) & (p <= hi) if i == 0 else (p > lo) & (p <= hi)
        if in_bin.sum():
            total += in_bin.sum() / len(y) * abs(y[in_bin].mean() - p[in_bin].mean())
    return total

def uniform_ece(y, p, bins):
    return binned_error(y, p, np.linspace(0, 1, bins + 1))

def adaptive_ece(y, p, bins):
    edges = np.quantile(p, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    return binned_error(y, p, edges)

averaged = {}
# Both are biased upward, and averaging over 40 replicates shows how each one grows.
for clustered in (False, True):
    for bins in (5, 15, 50):
        pairs = [(lambda t: (uniform_ece(*t, bins), adaptive_ece(*t, bins)))(
                    perfectly_calibrated(2000, seed=s, clustered=clustered))
                 for s in range(40)]
        u, a = np.mean([x[0] for x in pairs]), np.mean([x[1] for x in pairs])
        print(f"{'clustered' if clustered else 'spread':10} {bins:>2} {u:.4f} {a:.4f}")
# spread      5 0.0165 0.0164
# spread     15 0.0273 0.0275
# spread     50 0.0495 0.0491
# clustered   5 0.0088 0.0109
# clustered  15 0.0142 0.0187
# clustered  50 0.0253 0.0340

Two findings, both from a perfectly calibrated model averaged over forty replicates. On spread-out predictions the two strategies behave identically and both roughly triple as the bin count goes from five to fifty — adaptive binning does not reduce the upward bias and does not reduce sensitivity to \(M\). The mechanism is unchanged: more bins means fewer observations per bin, and fewer observations mean a larger chance deviation that the absolute value never lets cancel.

On clustered predictions — the case adaptive binning exists for — it reports a larger error than uniform binning at every bin count, by a quarter to a third. That is not a defect. Uniform binning pools nearly all the clustered predictions into one or two wide bins, where averaging cancels most of the noise and also hides any real structure inside them. Adaptive binning splits that crowded region into many small bins, each of which carries its own sampling error and each of which can show a distortion the wide bin was averaging away.

One caveat on the promise of equal counts. The bin edges come from quantiles of the predictions. Repeated scores can produce coincident edges, depending on the requested quantile positions and interpolation. Whether that costs you bins depends on how the ties are distributed, not just on how many distinct values there are. A vote share out of 100 trees has at most 101 distinct values, and asking for 50 bins on such a score is fine when the mass is spread across them — one observation at each of 0, 0.01, … , 1 fills all 50. It stops being fine when a repeated value swallows two or more of the requested quantile positions — a tied block cannot be split, so the edges around it coincide and bins are lost. Whether that happens depends on where the block sits relative to the quantile positions and on the interpolation rule, not on its size alone: 21 identical values out of 1,000 — over 2 percent — still leave all 51 edges distinct and all 50 bins occupied. In the extreme, if every prediction is the same number, every edge is that number and one bin holds everything. Report the number of bins actually realized, not the number requested.

Resolution, not accuracy

The right way to describe what adaptive binning buys is resolution where the predictions actually are. Where the scores are distinct the bins hold within one observation of each other, so no point on the resulting curve is built from far less evidence than the rest — though “the same amount” can still be very little: 150 distinct scores in 50 bins gives three observations each. Report the number of bins actually realized and their sizes, not just the number requested. The region a confident model spends its time in gets examined at a granularity proportional to how much data is there.

What it does not buy is a number that can be read as a verdict. The bias is still there, still grows with the bin count, and still means that a small adaptive ECE on a modest validation set is consistent with perfect calibration. The comparison that makes either variant interpretable is against a simulated perfect model on the same sample size and the same binning — which takes the few lines above and is almost never done.

The last row is a practical drawback that rarely gets mentioned. Because the boundaries are derived from the predictions, two models scored with adaptive binning are summarized over different partitions of the probability range, so the comparison is slightly less like-for-like than uniform binning gives. When comparing models or tracking one over time, inspect the realized boundaries and bin counts alongside the score. A change can reflect both calibration and the partition used to summarize it.

As with every binned summary, the diagnostic that carries more information than any of them is the reliability curve with bin counts shown, and the metric to use for model selection is a proper scoring rule such as the Brier score, which has no binning parameter at all.

References

• Chuan Guo, Geoff Pleiss, Yu Sun and Kilian Q. Weinberger, “On Calibration of Modern Neural Networks”, ICML 2017, arXiv:1706.04599
• scikit-learn documentation, sklearn.calibration.calibration_curve