Reliability Curves
A reliability curve — also called a calibration curve or reliability diagram — compares what a model predicted with what actually happened, across the range of its predictions. Predictions are grouped into bins; the horizontal position of each point is the average predicted probability within that bin, and the vertical position is the fraction of those cases that turned out positive. scikit-learn describes the vertical axis precisely as “an estimation of the conditional event probability \(P(Y=1 \mid \texttt{predict\_proba})\)”.
The population calibration relationship follows the diagonal y = x. Points estimated from finite data can depart from that line even when the model is calibrated. The plot helps locate possible problems along the probability range, but the size and uncertainty of each group determine how confidently a departure can be interpreted.
Reading the shape
| What the curve does | What it means | What tends to cause it |
|---|---|---|
| Sits below the diagonal everywhere | The stated probability is uniformly too high | A base rate that has shifted downward since training; a miscalibrated offset |
| Sits above the diagonal everywhere | The stated probability is uniformly too low | The same, in the other direction |
| S-shaped, crossing the diagonal near 0.5 | Overconfident if it runs below at the top and above at the bottom; underconfident if the reverse — see overconfident | Training to minimize cross-entropy on a flexible model, or heavy regularization in the other direction |
| Flat | Similar observed outcome rates across the chosen bins | Little observed separation between the chosen bins; also check binning and uncertainty |
| Wobbles around the diagonal | Possibly nothing at all | Sampling error — see below |
A worked reading starts with the counts. Suppose a bin contains 20 predictions whose probabilities sum to 15, and 12 outcomes are positive. Its plotted point is (15/20, 12/20) = (0.75, 0.60). The horizontal position is not the center of the interval containing those scores. At that point the observed event rate is 15 percentage points below the average prediction. Before treating the gap as systematic, inspect the small denominator and compare additional data.
For a larger illustrative pattern, points at (0.15, 0.05), (0.35, 0.20) and (0.75, 0.60) all suggest overprediction of the positive outcome. This differs from confidence in the chosen class: at a positive-class probability of 0.15, the predicted negative class has confidence 0.85, while negatives occur 95% of the time in the example. That is underconfidence in the chosen label. State whether the plot uses positive-class probability or top-label confidence. A fitted correction such as Platt scaling may help if the pattern persists; the curve alone does not identify its cause.
Every point is an estimate
The most common mistake with this plot is reading a single off-diagonal point as a finding. Each vertical coordinate is a proportion computed from however many cases fell into that bin, and proportions from small samples wander.
The uncertainty depends on counts and outcome frequencies. With independent observations, a rough standard error for an observed proportion r in a bin of n cases is √[r(1 − r)/n]. For r = 0.60 and n = 20 this is about 0.11, large relative to a 0.15 gap. This approximation is not a substitute for an appropriate interval, especially near zero or one or with dependent observations. Expected calibration error aggregates gaps and inherits this sampling uncertainty.
Display bin counts or a prediction histogram alongside the curve, and consider uncertainty for each point. A consistent pattern across sufficiently supported bins warrants investigation, but neither one off-diagonal point nor three consecutive points is a universal significance rule. Reusing the same data to select a calibration method and celebrate its improved curve can overstate the improvement.
How the bins are chosen
The curve is not a property of the model alone; it depends on a binning choice that the plot does not show.
| Strategy | How it splits | Behavior |
|---|---|---|
| Uniform width | Equal intervals — 0 to 0.1, 0.1 to 0.2, and so on | Covers the probability range evenly; bins in sparse regions may hold almost nothing |
| Quantile | Equal numbers of predictions per bin, where the predictions are distinct | Bin widths vary and a crowded region gets several narrow bins. Ties break the promise: repeated score values collapse the quantile edges, so the realized bins are unequal and some are empty — a vote share out of 100 trees is the common case |
For a model whose predictions pile up near zero — fraud, anomaly detection, rare-disease screening — uniform bins put nearly everything in the first bin and leave the upper bins with a handful of cases each. Quantile bins avoid that but compress the interesting high-probability region into one wide bin. Neither is right in general; the point is that changing the strategy changes the plot, so the strategy belongs in the caption.
Where the curve cannot help
- Regions with no predictions. The curve says nothing about probability ranges the model never outputs. A model that never predicts above 0.4 has no curve above 0.4, so calibration in that range cannot be assessed from this sample.
- Subgroups. An aggregate curve can sit on the diagonal while the model is overconfident for one segment and underconfident for another, because the two cancel. Plotting per segment is where the useful findings usually are — see model calibration.
- Ranking quality. A curve on the diagonal says the probabilities are honest, not that the model separates cases. A model predicting the base rate for everyone is calibrated on that population but unable to distinguish individual cases, and its curve is a single point on the line.
Read the curve alongside predictive performance and decision consequences. The Brier score evaluates probability accuracy; in its decomposition, reliability measures squared prediction–outcome gaps and resolution measures differences in outcome rates across forecast groups. The curve is a diagnostic view of a particular model, dataset and binning scheme, not a permanent certificate attached to a model name.
References
- scikit-learn User Guide, Probability calibration
- Chuan Guo, Geoff Pleiss, Yu Sun and Kilian Q. Weinberger, “On Calibration of Modern Neural Networks”, ICML 2017, arXiv:1706.04599
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.