Deep Ensembles

A deep ensemble combines the predictions of separately trained neural networks, often using the same architecture with different random initializations. For regression you average the outputs; for classification you average the predicted probabilities.

It is a useful baseline when assessing whether a more elaborate uncertainty method adds value. Lakshminarayanan, Pritzel and Blundell proposed it as an alternative to Bayesian networks that is “simple to implement, readily parallelizable, requires very little hyperparameter tuning” and reported uncertainty estimates as good as or better than approximate Bayesian methods.

What is averaged

For a classifier, each member supplies a probability vector over the same classes, and the ensemble averages corresponding probabilities. Three hypothetical positive-class probabilities of 0.9, 0.6, and 0.3 give 0.6. Averaging logits and applying sigmoid or softmax afterward generally gives a different result because those transformations are nonlinear. Keep class order and preprocessing consistent across members before combining outputs.

Different initializations and training randomness can produce different fitted functions even on the same data. Members need not have identical architectures, although using one architecture with different seeds provides a simple controlled baseline. Separate training runs do not make their errors statistically independent: shared data, features, objectives, and architectural biases can lead all members to the same mistake.

Why averaging can improve predictions

For a fixed example, log loss is convex in the probability assigned to the true class. Therefore the loss of an equal-weight probability average is no larger than the mean loss of those same members. This compares the ensemble with the average member loss, not with the best member. For a positive case with probabilities 0.99 and 0.01, the average probability is 0.5 and log loss is about 0.693; the better member has loss about 0.010. Adding a bad member can worsen an existing ensemble.

For regression, a simple variance calculation explains why correlated errors limit averaging. If errors have equal variance σ² and common pairwise correlation ρ, the variance of their M-member mean is σ²[ρ + (1 − ρ)/M]. With σ² = 1 and ρ = 0.5, four members give variance 0.625 and even an arbitrarily large ensemble approaches 0.5. Shared bias is not removed by averaging. This is a conditional model of the errors, not an estimate obtained just by counting seeds; assess gains on held-out data as members are added.

What disagreement means

Two ensembles can have the same average and different uncertainty signals. Members predicting [0.5, 0.5] agree that a binary outcome is ambiguous; members predicting [0.1, 0.9] disagree strongly, although both averages are 0.5. The first has no member spread despite an uncertain predictive outcome. Disagreement can indicate sensitivity to which fitted model is used, but it omits uncertainties all members share. All networks can confidently agree on the wrong answer.

For regression members that each predict a mean μₘ and a conditional variance vₘ, the equally weighted predictive mixture has mean μ̄ and variance equal to the average of vₘ plus the average of (μₘ − μ̄)². The first term captures uncertainty reported within each member; the second captures disagreement between their means. Two members with means 0 and 2 and variance 1 each give mixture mean 1 and variance 2. If members only output point estimates, their spread supplies no separate observation-noise estimate. The mixture also need not be Gaussian, so its mean and variance alone do not determine interval coverage.

Validate the intended use of uncertainty. For probabilities, check proper scores and calibration; for prediction intervals, inspect coverage and width. For a triage rule, measure retained-case error and how many cases are deferred at the chosen threshold. Test relevant shifts and segments separately. Active learning and unfamiliar-input detection can use disagreement, but neither is guaranteed to work just because multiple models are present.

What it costs, and what it is not

For M separately trained members, training requires M runs, storage generally includes M weight sets, and inference performs M forward passes. FLOPs per request therefore increase with the member count. Latency need not increase by the same factor: parallel execution, batching, device contention and scheduling determine the critical path. Measure latency and throughput under the intended resource budget. Model distillation can reduce deployment work, but matching the ensemble’s predictions does not by itself preserve its member-disagreement signal.

On whether deep ensembles are “Bayesian” the honest answer is that the label is contested and the question rarely matters operationally. They were introduced explicitly as a non-Bayesian method; others have since argued that averaging over independently trained modes approximates Bayesian model averaging better than a variational posterior around one mode does. What is not contested is the empirical record: in a large-scale comparison under dataset shift, Ovadia and colleagues found that “some methods that marginalize over models give surprisingly strong results” where post-hoc calibration and several other approaches fell short.

Snapshot ensembles reuse checkpoints from one training run; Monte Carlo dropout reuses a fitted network with different dropout masks at prediction time. They can reduce training or storage cost relative to separate full models, but their diversity and prediction cost still need measurement. Bayesian neural networks explicitly define a weight posterior through a prior and likelihood. Independently optimized ensemble members are not automatically samples from that posterior. Compare methods using the uncertainty decision and resource budget you actually need.

References

• Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2017). Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. arXiv:1612.01474. arxiv.org
• Ovadia, Y., et al. (2019). Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift. arXiv:1906.02530. arxiv.org
• scikit-learn documentation, sklearn.metrics.log_loss. scikit-learn.org