Model Distillation (Knowledge Distillation)

Definition

Model distillation is a model compression technique where a large, complex model (teacher) transfers its knowledge to a smaller, simpler model (student).

• Goal: create a lighter, faster model that retains most of the teacher’s performance.
• Widely used to deploy ML models on devices with limited resources (mobile, edge).

---

How It Works

1. Train a large teacher model
   • Example: a big transformer (BERT-large, GPT).
2. Generate soft targets
   • Instead of only using hard labels (0 or 1), the teacher provides probability distributions over classes.
   • Example: Teacher predicts [0.7 cat, 0.2 dog, 0.1 rabbit].
3. Train a smaller student model
   • Student is trained to mimic the teacher’s outputs (soft targets).
   • Loss function often combines:
     • Distillation loss (student vs. teacher probabilities).
     • Supervised loss (student vs. true labels).

---

Mathematical Form

$L = \alpha \cdot L_{\text{hard}}(y, p_s) \; + \; (1 – \alpha) \cdot L_{\text{soft}}(p_t, p_s, T)$

Where:

• $y$ = true labels
• $p_t$ = teacher probabilities
• $p_s$ = student probabilities
• $T$ = temperature (smooths teacher’s probabilities)
• $\alpha$ = balance factor

---

Why Use Distillation?

• Efficiency → smaller models run faster, cheaper inference.
• Deployment → suitable for edge devices (phones, IoT).
• Knowledge transfer → student can generalize better with teacher’s “dark knowledge” (soft probabilities).

---

Examples

1. NLP
   • DistilBERT: distilled from BERT → 40% smaller, 60% faster, ~97% of performance.
2. Vision
   • ResNet-50 distilled into ResNet-18 → similar accuracy, faster inference.
3. Speech / Audio
   • Distillation helps reduce latency for speech recognition models.

---

Benefits

• Smaller size (lower memory footprint).
• Faster inference (lower latency, cost).
• Retains much of teacher’s accuracy.

Challenges

• Student may not capture all teacher knowledge.
• Requires careful tuning of temperature & α.
• Still needs initial training of a large teacher model (expensive).

---

Summary
Model distillation = compressing a large model (teacher) into a smaller model (student) by training the student to mimic the teacher’s predictions.

• Saves cost and latency.
• Used widely in NLP (DistilBERT) and vision models.