Isotonic Regression

Isotonic regression fits the best non-decreasing function to data. Given pairs \((x_i, y_i)\) ordered by \(x\), it finds fitted values \(\hat{y}_1 \le \hat{y}_2 \le \dots \le \hat{y}_n\) minimizing \(\sum_i (y_i - \hat{y}_i)^2\). At the training points the solution is a step function: flat stretches joined by jumps. Between them it is a choice of the implementation — scikit-learn’s predict interpolates linearly between the fitted knots, so the function it applies to new scores is piecewise linear rather than piecewise constant.

The algorithm that finds it is pool adjacent violators. Walk along the sequence; wherever a value is lower than the one before it, merge the two blocks and give the merged block their weighted mean, weighted by how many original points each block already contains. Repeat until nothing decreases. Averaging the two block values without their weights gives the wrong answer as soon as a block holds more than one point. Each pooling step is the least-squares answer for that block, which is why the procedure is exact rather than approximate.

The last two observations violate monotonicity — the higher score had the lower outcome rate — so they are pooled into their common average of 0.525. The first two were already ordered and are left alone.

Why it is used for calibration

A model’s scores usually rank cases roughly correctly and state the wrong probabilities. Fixing that means finding a mapping from score to probability, and the one assumption worth making is that the mapping increases: a higher score should not mean a lower probability.

That is exactly the constraint isotonic regression imposes and the only one. scikit-learn puts the consequence plainly: “the only restriction is that the mapping function is monotonically increasing. It is thus more powerful as it can correct any monotonic distortion of the un-calibrated model.” Where Platt scaling can only express corrections of sigmoid shape, this can express all of them.

Like any calibrator it must be fitted on data the classifier did not train on, or it inherits the classifier’s optimism and produces probabilities “closer to 0 and 1 than it should”.

Sigmoid or isotonic, measured

The usual guidance is that isotonic wins given enough data — scikit-learn puts the threshold at around a thousand samples and warns that it “is more prone to overfitting, especially on small datasets”. The measurement below fits both calibrators on \(n\) points and scores them on twenty thousand held-out ones.

import numpy as np
from sklearn.isotonic import IsotonicRegression
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import brier_score_loss

def scores_and_labels(n, seed):
    """Scores that rank correctly but are distorted away from the true probability."""
    rng = np.random.default_rng(seed)
    p_true = rng.beta(2, 5, n)
    y = (rng.uniform(size=n) < p_true).astype(int)
    return y, np.clip(p_true ** 0.5, 1e-6, 1 - 1e-6)

# The calibrator is fitted on n_fit points and judged on 20,000 it has not seen.
for n_fit in (50, 100, 300, 1000, 5000, 20000):
    y, s = scores_and_labels(n_fit + 20000, seed=9)
    fit, test = slice(0, n_fit), slice(n_fit, n_fit + 20000)
    logit = np.log(s / (1 - s)).reshape(-1, 1)
    sigmoid = LogisticRegression().fit(logit[fit], y[fit]).predict_proba(logit[test])[:, 1]
    isotonic = IsotonicRegression(out_of_bounds="clip").fit(s[fit], y[fit]).predict(s[test])
    print(f"{n_fit:>6} {brier_score_loss(y[test], s[test]):.4f} "
          f"{brier_score_loss(y[test], sigmoid):.4f} {brier_score_loss(y[test], isotonic):.4f}")
# 50 0.2299 0.1924 0.1886
# 100 0.2296 0.1816 0.1858
# 300 0.2292 0.1817 0.1851
# 1000 0.2298 0.1801 0.1817
# 5000 0.2311 0.1795 0.1800
# 20000 0.2305 0.1790 0.1793

Two things stand out, and the second is more useful than the textbook rule. First, the gain from calibrating at all is large — the Brier score falls by about a fifth — while the difference between the two calibrators is in the third decimal place. Second, the sigmoid wins here at almost every size, including well above a thousand points.

That is not a contradiction of the guidance; it is a consequence of the distortion used. These scores were distorted by a square root, which is close to a sigmoid correction in logit space, so the parametric form fits it almost exactly and has nothing left for a flexible method to improve. On the differently distorted scores measured under Platt scaling, isotonic wins instead — 0.1808 against 0.1895. Which calibrator is better is a property of the distortion, not a general fact, and the cheap answer is to fit both and compare on held-out data.

What the step function costs

• It creates ties. Every score in a pooled block maps to the same probability, so cases that were strictly ordered become indistinguishable. That moves rank-based metrics slightly — AUC of 0.714483 becomes 0.713704 in the worked comparison — where a strictly monotone calibrator leaves them exactly unchanged. If the ordering is the product, that is a real if small cost.
• It is undefined outside the fitted range. A score higher than anything seen during calibration has no fitted value, so implementations require a policy — clipping to the end of the range is the usual one, and it means the model can no longer express anything above its calibration set’s maximum.
• It overfits when data is thin. With few points the steps follow noise, and each step is estimated from a handful of observations.
• It cannot fix a non-monotone distortion. If higher scores genuinely correspond to lower probabilities in some region, the constraint forces a flat stretch and the problem stays. That is a model problem, not a scaling problem.

Outside calibration

The same tool appears wherever a relationship is known to be monotone and its shape is not. Dose-response estimation assumes response does not fall as dose rises. Ranking and pricing problems often carry a monotonicity requirement for reasons of policy rather than statistics — a higher risk score must not produce a lower premium — and isotonic regression enforces it exactly while fitting the data as closely as that allows.

In every case the appeal is the same: one assumption, no functional form, and an exact solution. What it gives up is smoothness and extrapolation, which is why a fitted isotonic mapping is a lookup table rather than a formula. What calibration is for is covered under model calibration.

References

• scikit-learn User Guide, Probability calibration
• scikit-learn documentation, sklearn.isotonic.IsotonicRegression
• Alexandru Niculescu-Mizil and Rich Caruana, “Predicting Good Probabilities With Supervised Learning”, ICML 2005