Definition

An ensemble is a machine learning approach that combines multiple models to produce a single, stronger model.

  • Idea: a group of “weak” or diverse models can perform better together than individually.
  • Analogy: “wisdom of the crowd” → averaging many opinions gives a more accurate result.

Why Use Ensembles?

  • Reduce variance → more stable predictions (less sensitive to noise).
  • Reduce bias → capture more complex patterns.
  • Improve robustness → errors of one model can be corrected by others.
  • Often achieves state-of-the-art results in competitions (e.g., Kaggle).

Types of Ensembles

  1. Bagging (Bootstrap Aggregating)
    • Train many models on different random samples of the training data.
    • Combine predictions (usually by averaging or voting).
    • Example: Random Forest = ensemble of decision trees.
  2. Boosting
    • Train models sequentially; each new model focuses on correcting errors of the previous one.
    • Strong learners built from weak learners.
    • Examples: AdaBoost, Gradient Boosting, XGBoost, LightGBM, CatBoost.
  3. Stacking (Stacked Generalization)
    • Train multiple base models.
    • Use another model (meta-learner) to learn how to best combine their outputs.
    • Example: Logistic regression combining outputs of random forest + neural net.
  4. Voting Ensembles
    • Combine predictions of different models by:
      • Hard voting (majority class wins).
      • Soft voting (average predicted probabilities).
  5. Blending
    • Similar to stacking, but meta-learner trained on a holdout validation set, not cross-validation folds.

Examples

  • Kaggle Competitions: Winning teams often blend 5–20 models (XGBoost + LightGBM + neural nets).
  • Medical ML: Ensemble of CNNs for cancer detection improves reliability.
  • Fraud detection: Multiple tree models combined to reduce false negatives.

Advantages

  • Higher accuracy.
  • More robust against overfitting.
  • Can capture complementary patterns from different algorithms.

Disadvantages

  • More computationally expensive.
  • Harder to interpret (less explainable than a single model).
  • Deployment complexity (multiple models → higher latency, OpEx).

Summary
Ensembles = combining multiple models to improve accuracy, stability, and robustness.
Main strategies: Bagging, Boosting, Stacking, Voting, Blending.
They’re powerful but add cost and complexity.