Temperature Scaling

Temperature scaling divides a classifier’s logits by a single positive number \(T\) before the softmax:

\[ p_i = \frac{\exp(z_i / T)}{\sum_{j=1}^{K}\exp(z_j / T)} \]

At \(T = 1\) nothing changes. Above 1 the distribution flattens and the model sounds less certain; below 1 it sharpens. \(T\) is fitted on held-out data by minimizing negative log-likelihood, and it is the only thing fitted — one parameter for the whole model.

Guo and colleagues, who found that “modern neural networks, unlike those from a decade ago, are poorly calibrated”, describe it as “a single-parameter variant of Platt Scaling” and report that on most of the datasets they tried it is “surprisingly effective at calibrating predictions”. For a technique with one degree of freedom, that is a striking result, and it is the main reason the method is the default first thing to try on a network.

One parameter, and what that buys

Because the same \(T\) divides every logit, the ordering of the logits is untouched, so the largest one stays the largest. The predicted class cannot change — not usually, not approximately, but by construction.

import numpy as np

def softmax(logits, temperature=1.0):
    scaled = np.asarray(logits, dtype=float) / temperature
    scaled = scaled - scaled.max()
    return np.exp(scaled) / np.exp(scaled).sum()

logits = [2.0, 1.0, 0.1]
for t in (0.5, 1.0, 2.0, 5.0):
    p = softmax(logits, t)
    print(f"{t:<4} " + " ".join(f"{v:.4f}" for v in p) + f"  argmax={int(np.argmax(p))}")
# 0.5 0.8638 0.1169 0.0193 argmax=0
# 1.0 0.6590 0.2424 0.0986 argmax=0
# 2.0 0.5017 0.3043 0.1940 argmax=0
# 5.0 0.3996 0.3272 0.2733 argmax=0

From logits of 2.0, 1.0 and 0.1, raising the temperature moves the top probability from 0.86 down to 0.40 and the prediction never moves. What is preserved is the argmax within each sample, and therefore every accuracy figure and every confusion matrix. Ranking across samples is a different claim and it does not hold in the multi-class case: softmax renormalizes over the other classes, so two samples’ probabilities for the same class can swap order. With \(T = 0.5\), a sample at [0.6, 0.39, 0.01] and one at [0.59, 0.205, 0.205] exchange places on class 0. One-versus-rest AUC and any confidence-ordered queue can therefore move. What the method is safe for is leaving the predicted labels alone.

The same property is the binary-case invariance described under Platt scaling, where fitting a sigmoid leaves AUC identical to six decimal places. Calibration is a rescaling, not a re-decision.

What one parameter cannot do

• Class-dependent miscalibration. A single \(T\) is shared by every class, so a model that is overconfident on one class and underconfident on another cannot be fixed by it — the one parameter can only split the difference. Vector and matrix scaling add per-class parameters for exactly this, at the cost of more to fit and more to overfit.
• Non-monotone distortion. Temperature scaling is monotone in the logits, so a model whose confidence is wrong in a way that is not a uniform stretch stays wrong.
• Region-specific errors. A model well calibrated in the middle of its range and badly calibrated at the extremes needs something with local freedom, which is what isotonic regression provides.
• A distribution shift. \(T\) is fitted to one distribution. When the inputs move, the fitted temperature is as stale as the model, and a model that still ranks correctly can be badly miscalibrated again.

Fitting it

The temperature is chosen on data the network did not train on. Using training data produces the bias scikit-learn warns about for calibrators generally — “a biased calibrator that maps to probabilities closer to 0 and 1 than it should” — which here pushes the fitted temperature down, sharpening a model that was already overconfident. It does not guarantee \(T < 1\) — a network underconfident on its own training data can still fit a temperature above 1 — so treat training-set fitting as a source of optimistic bias rather than a predictable direction.

The standard objective is negative log-likelihood. It is convex in the inverse temperature \(1/T\), not in \(T\) — for a single logit gap of 1, \(L(T) = \log(1 + e^{-1/T})\) has second derivative about \(-0.3413\) at \(T = 1\). Optimize over \(1/T\), or accept that a search over \(T\) is a one-dimensional search over a function that need not be convex. Note that this optimizes log loss rather than expected calibration error, so the ECE can occasionally move the wrong way even as the calibration genuinely improves. Reporting a proper scoring rule alongside, and looking at a reliability curve, avoids being misled by either number alone.

A fitted \(T\) above 1 is the common outcome and indicates that the model was overconfident on the data used to fit it. The value is a uniform amount of softening, and it depends on the scale of the logits the network happens to produce, so there is no universal value at which it signals a problem. A large \(T\) is a prompt to look at the reliability curve and the errors themselves: if the network is confidently wrong on many validation cases, flattening everything treats the symptom.

Where it sits

Fewer parameters means less that can go wrong on a small validation set, and less that can be fixed. Starting at the top of that table and moving down only when the reliability curve shows a distortion the simpler method cannot absorb is the ordinary sequence.

References

• Chuan Guo, Geoff Pleiss, Yu Sun and Kilian Q. Weinberger, “On Calibration of Modern Neural Networks”, ICML 2017, arXiv:1706.04599
• scikit-learn User Guide, Probability calibration