Bayesian Decision Theory (BDT)

1. Core Idea

• Bayesian inference gives us a posterior distribution:
  • $p(\theta \mid D)$ where $\theta$ are unknown parameters and $D$ is observed data.
• But in practice, we don’t just want to know probabilities.
  We need to act (classify, predict, treat a patient, approve a loan, etc.).

Bayesian Decision Theory provides a principled way to make optimal decisions given uncertainty.

---

2. Components of Bayesian Decision Theory

A Bayesian decision problem has three ingredients:

1. Actions ($a$): possible choices you can take.
   Example: classify an email as spam or not spam.
2. States of nature ($\theta$): unknown truth of the world.
   Example: whether the email is actually spam or not.
3. Loss (or utility) function $L(a, \theta)$: penalty (or reward) for taking action $a$ when the true state is $\theta$.
   Example:
   • False positive (classify ham as spam) has high cost.
   • True positive (catch spam) has low/no cost.

---

3. Bayes Risk (expected loss)

Given data $D$, the posterior distribution is $p(\theta \mid D)$.
The Bayes risk of an action aaa is its expected loss:

$R(a \mid D) = \mathbb{E}_{\theta \sim p(\theta \mid D)}[L(a, \theta)]$

---

4. Bayes Optimal Decision Rule

The Bayes action is the one that minimizes posterior expected loss:

$a^*(D) = \arg\min_a R(a \mid D)$

This ensures you pick the action that, on average, does best under the posterior distribution.

---

5. Special Case: Classification

Suppose $\theta \in \{C_1, C_2, \dots, C_k\}$ are classes.

• Posterior class probabilities: $p(C_i \mid x)$.
• Loss: $L(a=C_j, \theta=C_i)$.

If 0–1 loss (correct = 0, incorrect = 1), then the Bayes optimal classifier is:

$a^*(x) = \arg\max_i \, p(C_i \mid x)$

This is exactly the MAP (maximum a posteriori) classifier.

---

6. Decision Thresholds with Asymmetric Loss

• If false positives and false negatives have different costs, BDT adjusts the decision boundary.
• Example (medical test):
  • Missing a disease (false negative) is worse than false alarm.
  • Decision threshold shifts → classify as positive at lower posterior probability.

---

7. Applications

• Machine Learning: classification, regression, Bayesian model selection.
• Medicine: treatment vs. no treatment decisions under uncertain diagnosis.
• Finance: portfolio choice under risk.
• Engineering: signal detection, robotics, reinforcement learning.

---

Summary:
Bayesian Decision Theory = Bayesian inference + decision-making.
It says: compute the posterior, define a loss function, then choose the action that minimizes expected loss (or maximizes expected utility).