Bayes’ Theorem

1. Definition

• Bayes’ Theorem is a rule in probability theory that describes how to update beliefs (probabilities) about a hypothesis or event when new evidence (data) is observed.
• It connects prior belief, likelihood of data, and posterior belief.

---

2. Formula

For a hypothesis $H$ and observed data $D$:

$P(H \mid D) = \frac{P(D \mid H) \cdot P(H)}{P(D)}$

Where:

• $P(H)$ = prior probability (belief before seeing data)
• $P(D \mid H)$ = likelihood (probability of data given the hypothesis)
• $P(D)$ = marginal probability of data (normalization)
• $P(H \mid D)$ = posterior probability (updated belief after seeing data)

---

3. Interpretation

• Prior ($P(H)$) → What we believed before seeing data.
• Likelihood ($P(D \mid H)$) → How compatible the data is with the hypothesis.
• Posterior ($P(H \mid D)$) → What we believe now, after seeing data.
• Evidence ($P(D)$) → Normalizing factor ensuring probabilities sum to 1.

In short:

$Posterior \propto Prior \times Likelihood$

---

4. Example – Medical Test

• Disease prevalence: 1% → $P(H) = 0.01$.
• Test sensitivity: 99% → $P(\text{Positive} \mid H) = 0.99$.
• Test false positive rate: 5% → $P(\text{Positive} \mid \neg H) = 0.05$.

Suppose a patient tests positive. What is the probability they actually have the disease?

$P(H \mid \text{Positive}) = \frac{P(\text{Positive} \mid H) P(H)}{P(\text{Positive})}$

Denominator:

$P(\text{Positive}) = 0.99 \cdot 0.01 + 0.05 \cdot 0.99 = 0.0099 + 0.0495 = 0.0594$

Posterior:

$P(H \mid \text{Positive}) = \frac{0.0099}{0.0594} \approx 0.167$

Even with a positive test, the probability of having the disease is ~16.7%, not 99%.

---

5. Applications of Bayes’ Theorem

• Statistics & Inference: Bayesian updating, posterior distributions.
• Machine Learning: Naive Bayes classifier, Bayesian networks.
• Medicine: Diagnostic tests, personalized treatment decisions.
• A/B Testing: Bayesian sequential testing, posterior probability of uplift.
• Finance & Risk: Fraud detection, portfolio risk estimation.
• Everyday reasoning: Updating beliefs when new information arrives.

---

6. Key Takeaways

• Bayes’ Theorem provides a formal way to update probabilities when new evidence arrives.
• Posterior = Prior × Likelihood / Evidence.
• Shifts belief from what we thought before (prior) to what we believe after seeing data (posterior).
• Crucial in uncertainty reasoning, A/B testing, and decision-making under incomplete information.

---

In short:
Bayes’ Theorem describes how to update probabilities of hypotheses given new data. It combines prior belief with the likelihood of evidence to produce the posterior probability.