Data Leakage

Definition

Data leakage happens when information from outside the training dataset sneaks into the model training process, giving the model unfair access to future or hidden knowledge.

• Leads to artificially high performance during training/validation.
• But performance drops sharply on real-world unseen data.

---

Types of Data Leakage

1. Target Leakage
   • Training data includes information that directly or indirectly contains the target label.
   • Example: predicting loan default using “debt_collected_after_default” as a feature → it already reveals the answer.
2. Train-Test Contamination
   • Test set information leaks into training (through preprocessing, feature scaling, etc.).
   • Example: scaling features using mean and std computed on the whole dataset instead of training set only.
3. Temporal Leakage
   • Using future data to predict the past/present.
   • Example: predicting stock price in January 2023 using features that include March 2023 trading volume.
4. Group Leakage
   • Same group (e.g., patient, user, session) appears in both train and test sets.
   • Model indirectly “remembers” patterns specific to that group.

---

Examples

• Medical AI: using “time of treatment outcome” as a feature when predicting disease diagnosis.
• Fraud detection: including “chargeback confirmed” in training features when predicting fraud.
• Cross-validation: forgetting to separate groups → patient appears in both train and validation folds.

---

How to Prevent Data Leakage

1. Data preprocessing properly
   • Compute normalization/scaling on training set only.
   • Apply same transformation to validation/test.
2. Feature selection carefully
   • Remove features that wouldn’t exist at prediction time.
3. Time-aware splits
   • For time series → train on past, validate/test on future.
4. Group-aware CV
   • Use GroupKFold or StratifiedGroupKFold to avoid overlap between train and validation.
5. Monitoring after deployment
   • If performance drops drastically compared to validation, check for hidden leakage.

---

Why It’s Dangerous

• Creates a false sense of model performance.
• Leads to overfitting and poor generalization.
• Can cause regulatory issues in sensitive domains (finance, healthcare).

---

Summary
Data leakage = when hidden or future information leaks into training, making the model unrealistically good during validation but bad in production.

• Types: target leakage, train-test contamination, temporal leakage, group leakage.
• Prevent with proper preprocessing, time-aware splits, and group-aware validation.