1) Definition

  • Deep Ensembles = train multiple deep neural networks independently (different initializations, sometimes different data subsets), and combine their predictions at inference.
  • Instead of relying on a single model, use an ensemble of models for better accuracy, robustness, and calibrated uncertainty.

Simple idea: “Don’t trust one deep net — trust the wisdom of many.”

2) How It Works

  1. Train $M$ neural networks:
    • Same architecture
    • Different random initialization
    • (Optionally) different bootstrapped data subsets
    • (Optionally) different hyperparameters
  2. At inference:
    • For regression: average predictions
      • $\hat{y} = \frac{1}{M}\sum_{m=1}^M f_m(x)$
    • For classification: average predicted probabilities
      • $P(y=k|x) = \frac{1}{M}\sum_{m=1}^M P_m(y=k|x)$

3) Why Use Deep Ensembles?

Better accuracy: Reduces variance and overfitting.
Robustness: More stable predictions under distribution shifts.
Uncertainty estimation: Variance across ensemble members can be used as an uncertainty measure (epistemic uncertainty).
Simplicity: Easy to implement with existing training pipelines.

4) Example

Suppose we train 5 independent ResNets on ImageNet.

  • Model 1 predicts: Cat (0.6), Dog (0.3), Fox (0.1)
  • Model 2 predicts: Cat (0.7), Dog (0.2), Fox (0.1)
  • Model 3 predicts: Cat (0.5), Dog (0.4), Fox (0.1)
  • …

Ensemble average: Cat (0.6), Dog (0.3), Fox (0.1) → more reliable.

If models disagree strongly, it signals uncertainty.

5) Applications

  • Classification/regression: standard tasks where robustness matters.
  • Medical AI: detect uncertainty in diagnoses.
  • Active learning: use ensemble disagreement to select new training points.
  • Out-of-distribution (OOD) detection: large variance across models signals OOD input.

6) Limitations

Compute cost: Training multiple deep nets = expensive.
Memory footprint: Need to store many models.
Not Bayesian: Provides uncertainty estimates, but not “true” Bayesian posterior.

7) Alternatives / Related Approaches

  • MC Dropout (approximate Bayesian inference).
  • Bayesian Neural Networks (but harder to scale).
  • Snapshot ensembles (train one model, save checkpoints at different epochs).
  • Distillation: compress ensemble into a single model.

Summary

  • Deep ensembles = multiple independently trained neural nets combined.
  • Improve accuracy, robustness, and uncertainty calibration.
  • Simple, strong baseline for uncertainty estimation.
  • Downsides: compute + memory heavy, but often worth it.