Underconfident

A prediction is underconfident when it expresses more uncertainty than is warranted by the relationship between predictions and outcomes. For example, among cases assigned 70% confidence in the chosen class, that class might be correct 90% of the time. This is the opposite direction from overconfidence. Probabilities near 0.5 are not enough to diagnose it: they may accurately describe difficult cases.

The signature

A simple binary example makes the direction visible. If predictions of 0.3 occur among cases whose event rate is 0.1, and predictions of 0.7 occur among cases whose event rate is 0.9, both predictions are too close to 0.5. On a positive-class reliability curve, the low point lies below the diagonal and the high point above it. The signed gap “prediction minus observed rate” is +0.2 at the low point and −0.2 at the high point.

This pattern illustrates an inward distortion, not a required shape for every model. Calibration errors can vary across regions or subgroups, with underconfidence in one region and overconfidence in another. Compute gaps among comparable predictions on held-out data and retain the denominators. Overall accuracy minus average confidence can conceal those opposing errors.

Caution is not free

A strictly proper scoring rule rewards truthful probability reporting in expectation. If the true event probability is 0.9, reporting 0.7 has expected Brier loss 0.9(0.7 − 1)² + 0.1(0.7 − 0)² = 0.13. Reporting 0.9 gives 0.09. The excess 0.04 is the squared distortion (0.7 − 0.9)². This is a consequence of the assumed true probability, not a claim that one forecast always wins in a finite sample.

Sharpness describes how concentrated a predictive distribution is, independently of observed outcomes. Calibration asks whether the distribution agrees with outcomes. Gneiting, Balabdaoui and Raftery advocate pursuing sharpness subject to calibration: confident predictions are desirable when justified. A constant base-rate forecast can be calibrated yet offer little case-level information; that is low resolution rather than proof of underconfidence.

Underconfidence can change actions when a decision uses probability values. A case with estimated probability 0.7 will not pass a cutoff of 0.8 even if the relevant conditional event rate is 0.9. That can miss valuable interventions or needed reviews. The size and direction of the cost depend on the decision, so neither “caution is safe” nor “more extreme is better” is a general rule.

Wide is not the same as underconfident

Two different things get described as intervals that are too wide, and only one of them is underconfidence.

A wide interval may correctly reflect limited data, outcome variability or model uncertainty. Width alone cannot show that it is too conservative. Assess a confidence procedure through its repeated-sample coverage under stated conditions, and assess predictive intervals against future outcomes. Narrowing an interval without evidence can replace apparent caution with undercoverage.

A nominal 95% procedure with 99.9% coverage under an evaluated setting is conservative there. Extra coverage can result from approximations, discreteness or deliberate simultaneous-coverage requirements. It does not automatically establish that a narrower interval meets the intended guarantee across all required settings. Compare procedures with the same target coverage and scope before attributing the lost precision to an error.

Where it comes from

• Heavy regularization. An L1 or L2 penalty strong enough to shrink coefficients toward zero shrinks the logits with them, and the probabilities collapse toward the base rate.
• Averaging over many models. Averaging can pull extreme predictions inward when members disagree. That may improve calibration or create underconfidence; only outcome-based evaluation can distinguish the two.
• Model classes that hedge by construction. Averaging tree probabilities can make extreme outputs less common. This is a possible calibration mechanism, not proof that all forests are underconfident.
• A vague prior in a Bayesian model. Prior information and model assumptions affect uncertainty, but a vague prior does not by itself diagnose underconfidence. Check the resulting posterior or predictive behavior under the intended evaluation.
• Deliberate hedging in a high-stakes setting. A model tuned to avoid extreme outputs in medicine or finance. This is a real motivation, but it puts the caution in the wrong place — see below.

Fixing it

If underconfidence is a strictly increasing distortion of an otherwise informative scalar probability, ranking is preserved. In the exact Murphy decomposition of the Brier score, grouping by distinct forecast value also preserves resolution under such a transformation. That conditional result does not mean every underconfident model retains all useful information. Missing features, subgroup errors and poor discrimination may require changes to the underlying model.

So the repair is the same machinery as for overconfidence, running the other way. Platt scaling fits an S-shaped correction and can stretch probabilities outward as readily as it compresses them. Temperature scaling with a positive temperature below 1 makes logits more extreme; whether that improves the model must be checked on held-out data. Isotonic regression handles a distortion of any monotone shape. Fit on held-out data in every case.

When limited discrimination or missing information accompanies a calibration problem, compare a calibration correction with changes to regularization, features or model structure. Validate the resulting probabilities and decisions; stretching predictions cannot supply information the model never learned. And if the caution was deliberate — a decision not to act on thin evidence — it belongs in the decision rule rather than the probability. Keep the probabilities honest and move the cut point, as under recalibrate thresholds.

References

• Tilmann Gneiting, Fadoua Balabdaoui and Adrian E. Raftery, “Probabilistic Forecasts, Calibration and Sharpness”, Journal of the Royal Statistical Society Series B, 69(2), 2007, 243–268
• scikit-learn User Guide, Probability calibration