Definition
k-fold cross-validation is a technique where the dataset is split into k equally (or nearly equally) sized folds, and the model is trained/tested k times, each time using a different fold as the test set.
- It’s the default cross-validation method used in most ML tasks.
- Helps reduce variance in performance estimation compared to a single train/test split.
How It Works
- Shuffle dataset (if order doesn’t matter).
- Split into k folds (e.g., k = 5 or 10).
- For each fold $i$:
- Train on $k-1$ folds.
- Test on the remaining 1 fold.
- Collect performance scores from each test.
- Compute average score → final evaluation metric.
Example (k = 5)
Dataset = 1,000 samples → 5 folds (200 samples each).
- Fold 1: Train on 800, validate on 200.
- Fold 2: Train on 800, validate on 200.
- … repeat until all 5 folds used for testing.
- Final result = mean accuracy across 5 runs.
Variations
- Stratified k-Fold → ensures class proportions remain the same across folds (important for imbalanced datasets).
- Repeated k-Fold → repeat CV multiple times with different splits → more robust estimate.
- Leave-One-Out CV (LOOCV) → extreme case: k = N (one sample per fold).
Why Use It
- More robust evaluation (less variance than a single test split).
- Makes better use of data (all samples used for both training & validation).
- Essential for hyperparameter tuning (grid search, random search).
Pros & Cons
Pros
- Reliable generalization estimate.
- Reduces bias from one random split.
- Works for most datasets.
Cons
- Computationally expensive (train model k times).
- Not always ideal for time-series data (need time-aware CV instead).
Typical k Values
- k = 5 or k = 10 → most common in practice.
- k = N (LOOCV) → very accurate but very expensive.
Summary
k-fold cross-validation = splitting data into k parts, training k times with different test folds, and averaging results.
- Provides a robust estimate of model performance.
- Commonly used in model evaluation & hyperparameter tuning.