Weak Supervision
1) Definition
- Weak supervision = using imperfect, noisy, or indirect labels instead of (or in addition to) clean, manually annotated labels.
- Goal: build models when labeled data is limited or expensive.
- Labels may come from heuristics, rules, external knowledge bases, or multiple annotators.
Instead of spending $$$ on perfect labels, we use weaker signals to train.
2) Sources of Weak Labels
- Heuristic rules
- e.g., If an email contains “free $$$”, label as spam.
- Distant supervision
- Use external knowledge base to label automatically.
- Example: If a text mentions “Barack Obama” and “USA”, label relation = PresidentOf.
- Crowdsourcing / Multiple annotators
- Each annotator may be noisy → aggregate with majority vote.
- Programmatic labeling (Snorkel, etc.)
- Write labeling functions (LFs) to generate weak labels at scale.
- Data augmentation
- Synthetic labels from transformations (back-translation, image rotation).
3) Why use Weak Supervision?
- Manual labeling is expensive, slow, and often infeasible at scale.
- Weak supervision lets you leverage large unlabeled datasets cheaply.
- In many real-world tasks (fraud, medical, NLP), weak supervision can be the only practical option.
4) Challenges
- Labels are noisy → risk of training on errors.
- Class imbalance may be amplified.
- Requires denoising or combining weak labels intelligently.
5) Approaches to Handle Noise
- Label model (aggregation): combine multiple weak sources into a consensus label (e.g., Snorkel).
- Robust learning: train with loss functions that tolerate noise (MAE, generalized cross-entropy).
- Confidence weighting: weight examples by estimated label reliability.
- Semi-supervised hybrid: mix small clean dataset with large weakly labeled dataset.
6) Example
Suppose we want to classify tweets as toxic / non-toxic, but we can’t label millions by hand.
- Weak label sources:
- Heuristic: tweets with “idiot” → toxic.
- Distant supervision: match toxic word list.
- Crowdsourcing: multiple annotators with disagreement.
- Aggregation: combine sources → produce probabilistic labels.
- Train model: fine-tune BERT using aggregated weak labels.
- Result: better than training on a tiny set of gold labels only.
7) Applications
- NLP: entity/relation extraction, sentiment analysis.
- Vision: pseudo-labeling, self-training.
- Healthcare: auto-label from clinical codes (ICD), then refine.
- Search/recommendation: click data as a weak label for relevance.
Summary
- Weak supervision = training with noisy/incomplete labels instead of perfect ones.
- Sources: heuristics, knowledge bases, multiple annotators, programmatic labeling.
- Benefits: scales cheaply, unlocks massive unlabeled datasets.
- Requires noise-handling (aggregation, robust loss, semi-supervised approaches).