Adaptive ECE (Expected Calibration Error with Adaptive Binning)

ByJu Yeon Eum

Background: Standard ECE

  • ECE (Expected Calibration Error) measures how well predicted probabilities match observed frequencies.
  • Standard ECE uses fixed bins (e.g., [0.0–0.1], [0.1–0.2], …).
  • Problem: If predictions are not uniformly distributed across probability space, some bins may contain very few or even zero samples → noisy estimates.

Adaptive Binning (Motivation)

  • Instead of dividing the probability space into equal width bins, adaptive binning divides it into equal count bins.
  • Each bin has roughly the same number of samples, which reduces variance in estimates and gives more stable calibration error measurement.

Formal Definition

  1. Sort all predictions $\hat{p}_i$ in ascending order.
  2. Divide them into $M$ bins $B_1, B_2, \dots, B_M$​, each with about $N/M$ predictions (where NNN is total number of samples).
    • This means bin widths vary depending on where predictions cluster.
  3. For each bin $B_m$​, compute:
    • Confidence:
      • $\text{conf}(B_m) = \frac{1}{|B_m|} \sum_{i \in B_m} \hat{p}_i$
    • Accuracy:
      • $\text{acc}(B_m) = \frac{1}{|B_m|} \sum_{i \in B_m} \mathbf{1}(\hat{y}_i = y_i)$
  4. Adaptive ECE is:
    • $\text{Adaptive ECE} = \sum_{m=1}^M \frac{|B_m|}{N} \; \big| \text{acc}(B_m) – \text{conf}(B_m) \big|$

Intuition

  • Standard ECE: Fixed bins → some bins may be empty or noisy.
  • Adaptive ECE: Balanced bins → more stable and fair evaluation, especially when predictions cluster (e.g., most probabilities between 0.8 and 1.0).
  • This makes Adaptive ECE less sensitive to the choice of $M$.

Example

Suppose you have 1,000 predictions, and you want 10 bins:

  • Standard ECE: Each bin covers width 0.1 in probability space. Some bins (like [0.0–0.1]) may have very few points if your model rarely predicts low probabilities.
  • Adaptive ECE: Each bin has 100 predictions, even if that means bin boundaries are uneven (e.g., one bin might cover [0.0–0.6], the next [0.6–0.75], etc.).

This ensures every bin’s accuracy/confidence is estimated from enough samples.

Advantages

  • Handles class imbalance and skewed probability distributions better.
  • Produces lower variance estimates of calibration error.
  • More representative of how well the model is calibrated across the range it actually uses.

Limitations

  • Bin boundaries are data-dependent, so results are not directly comparable across different datasets or runs.
  • May hide local calibration problems if the number of bins $M$ is too small.

Relation to Other Metrics

  • ECE (fixed bins): Easier to interpret, but unstable if bins are sparse.
  • Adaptive ECE: More reliable for skewed predictions.
  • MCE (Maximum Calibration Error): Shows worst-case gap, not average.
  • Brier Score: Measures both calibration and discrimination (squared error of probabilities).

Summary:
Adaptive ECE = ECE with adaptive binning, where bins are constructed to contain equal numbers of samples instead of equal probability widths. This improves stability of calibration error estimates, especially when predictions are skewed.

Similar Posts

Leave a Reply