Cross-Validation (CV)

Definition

Cross-validation is a model evaluation technique that splits data into multiple subsets (folds) to ensure that performance estimates are robust and not overly dependent on a single train/test split.

• Goal: test how well a model generalizes to unseen data.
• Helps with model selection, hyperparameter tuning, and overfitting prevention.

---

How It Works

1. Split dataset into k folds (subsets).
2. For each fold:
   • Train model on k–1 folds.
   • Test model on the remaining fold.
3. Repeat k times (each fold is used as test once).
4. Average performance across folds → final estimate.

---

Common Types of Cross-Validation

1. k-Fold Cross-Validation
   • Data split into k folds (e.g., k=5 or 10).
   • Balanced trade-off between bias & variance.
2. Stratified k-Fold CV
   • Keeps class distribution consistent across folds (important for imbalanced classification).
3. Leave-One-Out Cross-Validation (LOOCV)
   • Each fold has exactly 1 sample.
   • Very accurate but computationally expensive.
4. Time-Series Cross-Validation (Rolling/Expanding Window)
   • For time-dependent data (can’t shuffle).
   • Train on past → validate on future.
5. Nested Cross-Validation
   • Outer loop: model evaluation.
   • Inner loop: hyperparameter tuning.
   • Prevents overfitting during model selection.

---

Why It’s Important

• Reduces variance from one random train/test split.
• Ensures robust model evaluation.
• Helps tune hyperparameters (via grid/random/Bayesian search with CV).

---

Example

Dataset = 1,000 samples, 5-fold CV:

• Fold 1: Train on 800, validate on 200.
• Fold 2: Train on 800, validate on 200.
• … repeat until all folds used.
• Final score = mean accuracy across 5 folds.

---

Pros & Cons

Pros

• More reliable estimate of model performance.
• Reduces overfitting to one test set.
• Useful for hyperparameter tuning.

Cons

• Computationally expensive (train model k times).
• Must choose fold type carefully (esp. time-series).

---

Summary
Cross-validation = evaluating a model by splitting data into multiple folds and rotating train/test sets.

• Types: k-fold, stratified, LOOCV, time-series, nested.
• Goal: robust, unbiased estimate of generalization performance.