Ensemble

Definition

An ensemble is a machine learning approach that combines multiple models to produce a single, stronger model.

• Idea: a group of “weak” or diverse models can perform better together than individually.
• Analogy: “wisdom of the crowd” → averaging many opinions gives a more accurate result.

---

Why Use Ensembles?

• Reduce variance → more stable predictions (less sensitive to noise).
• Reduce bias → capture more complex patterns.
• Improve robustness → errors of one model can be corrected by others.
• Often achieves state-of-the-art results in competitions (e.g., Kaggle).

---

Types of Ensembles

1. Bagging (Bootstrap Aggregating)
   • Train many models on different random samples of the training data.
   • Combine predictions (usually by averaging or voting).
   • Example: Random Forest = ensemble of decision trees.
2. Boosting
   • Train models sequentially; each new model focuses on correcting errors of the previous one.
   • Strong learners built from weak learners.
   • Examples: AdaBoost, Gradient Boosting, XGBoost, LightGBM, CatBoost.
3. Stacking (Stacked Generalization)
   • Train multiple base models.
   • Use another model (meta-learner) to learn how to best combine their outputs.
   • Example: Logistic regression combining outputs of random forest + neural net.
4. Voting Ensembles
   • Combine predictions of different models by:
     • Hard voting (majority class wins).
     • Soft voting (average predicted probabilities).
5. Blending
   • Similar to stacking, but meta-learner trained on a holdout validation set, not cross-validation folds.

---

Examples

• Kaggle Competitions: Winning teams often blend 5–20 models (XGBoost + LightGBM + neural nets).
• Medical ML: Ensemble of CNNs for cancer detection improves reliability.
• Fraud detection: Multiple tree models combined to reduce false negatives.

---

Advantages

• Higher accuracy.
• More robust against overfitting.
• Can capture complementary patterns from different algorithms.

Disadvantages

• More computationally expensive.
• Harder to interpret (less explainable than a single model).
• Deployment complexity (multiple models → higher latency, OpEx).

---

Summary
Ensembles = combining multiple models to improve accuracy, stability, and robustness.
Main strategies: Bagging, Boosting, Stacking, Voting, Blending.
They’re powerful but add cost and complexity.