KS Statistic (Kolmogorov–Smirnov Statistic)

1) Meaning

The KS statistic is a measure of the maximum difference between two cumulative distribution functions (CDFs).

• In statistics: used to test if two samples come from the same distribution (KS test).
• In machine learning (especially credit scoring / binary classification): the KS statistic measures how well a model separates positive vs. negative classes.

Intuition:

• If the two distributions are very different → KS is large.
• If they overlap heavily → KS is small.

---

2) Formula

For binary classification (positive = 1, negative = 0):

1. Compute CDF of positives: proportion of actual positives up to a given score threshold.
2. Compute CDF of negatives: proportion of actual negatives up to the same threshold.
3. KS statistic = maximum vertical distance between the two CDFs:

$KS = \max_x \; | F_{\text{positive}}(x) – F_{\text{negative}}(x) |$

Where $F$ = cumulative distribution.

---

3) Example

Suppose we score 1,000 customers with a credit risk model:

• 500 are “good” (non-default), 500 are “bad” (default).
• Sort customers by predicted score.
• At each threshold, compute:
  • % of bads captured (True Positive Rate)
  • % of goods captured (False Positive Rate)
• KS = largest gap between these two curves.

If at score = 0.65:

• CDF bads = 0.70 (70% of bads identified)
• CDF goods = 0.30 (30% of goods misclassified)
• Gap = 0.40 → KS = 40%

---

4) Interpretation

• KS ranges from 0 → 1 (or 0% → 100%).
  • Higher KS = better separation.
  • KS = 0 → model has no power (both distributions identical).
  • KS ≈ 0.4–0.6 → strong discriminatory power (common in credit risk models).
• In practice:
  • KS < 0.2 → weak model.
  • KS ~ 0.3–0.4 → moderate.
  • KS > 0.4 → strong.

---

5) Applications

• Credit Scoring: KS is one of the most used model evaluation metrics.
• Hypothesis Testing: KS test compares sample vs. reference distribution.
• Model Validation: Detects overfitting or poor generalization if KS differs greatly between train and test sets.

---

6) Relation to Other Metrics

• Similar to AUC (ROC curve): both measure separability of classes.
• AUC integrates the whole ROC curve, while KS focuses on the maximum point of separation.
• Both are threshold-independent metrics.

---

Bottom line:
The KS statistic measures the maximum difference between cumulative distributions of two groups (positives vs. negatives). In ML, a higher KS means better class separation, making it a key metric in credit scoring and binary classification evaluation.