Predictive Parity (Calibration)

1. Definition

• Predictive Parity (aka Calibration) means that, for individuals who receive the same predicted score or probability, the actual outcomes (true positive rate) should be the same across demographic groups.

Formally, for a predicted probability $\hat{p}$​:

$P(Y=1 \mid \hat{P}=\hat{p}, A=a) = P(Y=1 \mid \hat{P}=\hat{p}, A=b) \quad \forall a,b$

Where:

• $Y$ = true outcome
• $\hat{P}$ = model’s predicted probability
• $A$ = protected attribute (e.g., gender, race)

In plain terms: If two people from different groups are both predicted to have a 70% chance of success, then roughly 70% of each group should actually succeed.

---

2. Example

Loan approval model:

• For people with a predicted 0.7 default probability:
  • Group A: 70% actually default.
  • Group B: 50% actually default.

The model is not calibrated across groups, violating predictive parity.

If both groups had 70% actual default at score 0.7, predictive parity would hold.

---

3. Why It’s Important

• Ensures that predicted probabilities are interpretable and fair across groups.
• Without calibration, risk scores may overestimate or underestimate risk for certain groups.
• Useful in credit scoring, recidivism prediction, healthcare risk models.

---

4. Relation to Other Fairness Metrics

• Demographic Parity → focuses on equal prediction rates (ignores labels).
• Equal Opportunity / Equalized Odds → focuses on error rates (TPR/FPR).
• Predictive Parity (Calibration) → focuses on predicted probability consistency across groups.

These criteria can conflict:

• If base rates differ across groups, you usually cannot satisfy Equalized Odds and Predictive Parity at the same time (this is known as the fairness impossibility theorem).

---

5. Limitations

• Calibration does not guarantee fairness in errors.
  • A model may be calibrated but still deny more opportunities to one group.
• Hard to satisfy simultaneously with other fairness definitions.
• Sensitive to sample size (small groups may look uncalibrated due to variance).

---

Summary:
Predictive Parity (Calibration) = predicted probabilities mean the same thing across groups.
If two people (from any group) get a 70% predicted chance, then ~70% of them should actually succeed.