k-fold cross-validation

Definition

k-fold cross-validation is a technique where the dataset is split into k equally (or nearly equally) sized folds, and the model is trained/tested k times, each time using a different fold as the test set.

• It’s the default cross-validation method used in most ML tasks.
• Helps reduce variance in performance estimation compared to a single train/test split.

---

How It Works

1. Shuffle dataset (if order doesn’t matter).
2. Split into k folds (e.g., k = 5 or 10).
3. For each fold $i$:
   • Train on $k-1$ folds.
   • Test on the remaining 1 fold.
4. Collect performance scores from each test.
5. Compute average score → final evaluation metric.

---

Example (k = 5)

Dataset = 1,000 samples → 5 folds (200 samples each).

• Fold 1: Train on 800, validate on 200.
• Fold 2: Train on 800, validate on 200.
• … repeat until all 5 folds used for testing.
• Final result = mean accuracy across 5 runs.

---

Variations

• Stratified k-Fold → ensures class proportions remain the same across folds (important for imbalanced datasets).
• Repeated k-Fold → repeat CV multiple times with different splits → more robust estimate.
• Leave-One-Out CV (LOOCV) → extreme case: k = N (one sample per fold).

---

Why Use It

• More robust evaluation (less variance than a single test split).
• Makes better use of data (all samples used for both training & validation).
• Essential for hyperparameter tuning (grid search, random search).

---

Pros & Cons

Pros

• Reliable generalization estimate.
• Reduces bias from one random split.
• Works for most datasets.

Cons

• Computationally expensive (train model k times).
• Not always ideal for time-series data (need time-aware CV instead).

---

Typical k Values

• k = 5 or k = 10 → most common in practice.
• k = N (LOOCV) → very accurate but very expensive.

---

Summary
k-fold cross-validation = splitting data into k parts, training k times with different test folds, and averaging results.

• Provides a robust estimate of model performance.
• Commonly used in model evaluation & hyperparameter tuning.