Bayes’ Theorem: Reversing Conditional Probabilities

Bayes’ theorem provides a systematic way to update probabilities after observing new evidence. It is especially useful when the conditional probability we know is the reverse of the probability we need.

In an email-filtering example, we may know:\[ P(\text{Money}\mid\text{Spam}), \]

the probability that a spam email contains the word “money.”

However, a classifier needs:\[ P(\text{Spam}\mid\text{Money}), \]

the probability that an email is spam after observing that it contains the word.

These probabilities are related, but they are not equal. Bayes’ theorem connects them.

Defining the Events

Let:\[ S=\text{the email is spam} \]\[ H=\text{the email is legitimate} \]\[ M=\text{the email contains the word “money”}. \]

Assume spam and legitimate email are mutually exclusive and exhaustive:\[ H=S^c. \]

Suppose the following illustrative probabilities are known:\[ P(M\mid S)=0.08 \]\[ P(M\mid H)=0.01 \]\[ P(S)=0.20. \]

Therefore:\[ P(H)=1-P(S)=0.80. \]

The goal is to calculate:\[ P(S\mid M). \]

The Direction of Conditioning Matters

The probability:\[ P(M\mid S)=0.08 \]

means:

Among spam emails, 8% contain “money.”

The probability:\[ P(S\mid M) \]

means:

Among emails containing “money,” what percentage are spam?

The denominators differ.

Using counts:\[ P(M\mid S) = \frac{ \text{spam emails containing “money”} }{ \text{all spam emails} } \]

while:\[ P(S\mid M) = \frac{ \text{spam emails containing “money”} }{ \text{all emails containing “money”} }. \]

Because the reference groups are different:\[ P(M\mid S)\neq P(S\mid M) \]

in general.

Deriving Bayes’ Theorem

The definition of conditional probability gives:\[ P(B\mid A) = \frac{P(A\cap B)}{P(A)}. \]

The joint event can be written in either order:\[ A\cap B=B\cap A. \]

The general multiplication rule gives:\[ P(A\cap B) = P(B)P(A\mid B). \]

Substituting into the conditional-probability formula:\[ P(B\mid A) = \frac{ P(B)P(A\mid B) }{ P(A) }. \]

Therefore:\[ \boxed{ P(B\mid A) = \frac{ P(A\mid B)P(B) }{ P(A) } } \]

This is Bayes’ theorem.

An equivalent form is:\[ \boxed{ P(A\mid B) = \frac{ P(B\mid A)P(A) }{ P(B) } } \]

The formula does not merely swap the two events. It also adjusts by their marginal probabilities.

Applying Bayes’ Theorem to Spam

We want:\[ P(S\mid M). \]

Bayes’ theorem gives:\[ P(S\mid M) = \frac{ P(M\mid S)P(S) }{ P(M) }. \]

We already know:\[ P(M\mid S)=0.08 \]

and:\[ P(S)=0.20. \]

The remaining quantity is:\[ P(M), \]

the overall probability that a randomly selected email contains the word.

Calculating the Denominator

The event \(M\) can occur through two mutually exclusive paths:

the email is spam and contains the word;
the email is legitimate and contains the word.

Therefore:\[ M=(M\cap S)\cup(M\cap H). \]

Using the law of total probability:\[ P(M) = P(M\mid S)P(S) + P(M\mid H)P(H). \]

Substitute the values:\[ P(M) = (0.08)(0.20) + (0.01)(0.80). \]

Calculate each branch:\[ (0.08)(0.20)=0.016 \]

and:\[ (0.01)(0.80)=0.008. \]

Therefore:\[ P(M) = 0.016+0.008 = 0.024. \]

Thus, 2.4% of all emails contain the word under this model.

Calculating the Posterior Probability

Now substitute into Bayes’ theorem:\[ P(S\mid M) = \frac{ (0.08)(0.20) }{ 0.024 }. \]

Therefore:\[ P(S\mid M) = \frac{0.016}{0.024} = \frac{2}{3}. \]

As a percentage:\[ \boxed{ P(S\mid M)\approx66.7\% } \]

Under the illustrative assumptions, an email containing “money” has approximately a 67% probability of being spam.

The Expanded Form of Bayes’ Theorem

When the denominator is not given directly, substitute the law of total probability:\[ P(M) = P(M\mid S)P(S) + P(M\mid H)P(H). \]

This produces:\[ \boxed{ P(S\mid M) = \frac{ P(M\mid S)P(S) }{ P(M\mid S)P(S) + P(M\mid H)P(H) } } \]

Numerically:\[ P(S\mid M) = \frac{ (0.08)(0.20) }{ (0.08)(0.20)+(0.01)(0.80) }. \]

This is the two-class expanded form of Bayes’ theorem.

A Frequency-Table Interpretation

Probabilities are often easier to understand by converting them into hypothetical counts.

Suppose there are 10,000 emails.

Because:\[ P(S)=0.20, \]

the expected numbers are:\[ 10{,}000(0.20)=2{,}000 \]

spam emails and:\[ 10{,}000(0.80)=8{,}000 \]

legitimate emails.

Of the 2,000 spam emails:\[ 2{,}000(0.08)=160 \]

contain “money.”

Of the 8,000 legitimate emails:\[ 8{,}000(0.01)=80 \]

contain it.

Email type	Contains “money”	Does not contain it	Total
Spam	160	1,840	2,000
Legitimate	80	7,920	8,000
Total	240	9,760	10,000

Among the 240 emails containing “money,” 160 are spam:\[ P(S\mid M) = \frac{160}{240} = \frac{2}{3}. \]

This gives the same 66.7% result.

A Probability-Tree Interpretation

The calculation can also be organized as a tree:

			
Email
├── Spam: 0.20
│   ├── Money: 0.08      → 0.016
│   └── No money: 0.92   → 0.184
└── Legitimate: 0.80
    ├── Money: 0.01      → 0.008
    └── No money: 0.99   → 0.792

		

Multiply along each branch:\[ P(S\cap M) = P(S)P(M\mid S) = 0.016. \]

Then add the branches compatible with the observed evidence:\[ P(M) = 0.016+0.008 = 0.024. \]

Finally, calculate the fraction of the evidence attributable to spam:\[ P(S\mid M) = \frac{0.016}{0.024}. \]

Prior, Likelihood, Evidence, and Posterior

Bayes’ theorem is often written using four terms:\[ \boxed{ \text{Posterior} = \frac{ \text{Likelihood}\times\text{Prior} }{ \text{Evidence} } } \]

For the spam example:

Prior

\[ P(S)=0.20 \]

The prior is the probability of spam before inspecting the word.

Likelihood

\[ P(M\mid S)=0.08 \]

The likelihood describes how compatible the observed word is with spam.

Evidence

\[ P(M)=0.024 \]

The evidence is the overall probability of observing the word under all possible email classes.

Posterior

\[ P(S\mid M)=0.667 \]

The posterior is the updated probability after observing the word.

The update can be summarized as:\[ 20\% \quad\xrightarrow{\text{observe “money”}}\quad 66.7\%. \]

Why the Posterior Is Much Larger Than the Likelihood

It may initially seem surprising that:\[ P(M\mid S)=8\% \]

but:\[ P(S\mid M)\approx67\%. \]

There is no contradiction.

Only a minority of spam emails contain the word, so the first probability is relatively small. But the word is much rarer in legitimate email:\[ P(M\mid H)=1\%. \]

Consequently, when the word does appear, it provides evidence favoring spam.

The relevant comparison is:\[ \frac{ P(M\mid S) }{ P(M\mid H) } = \frac{0.08}{0.01} = 8. \]

The word is eight times as likely under spam as under legitimate email.

Likelihood Ratio

The quantity:\[ \operatorname{LR}_M = \frac{ P(M\mid S) }{ P(M\mid H) } \]

is the likelihood ratio associated with observing \(M\).

In this example:\[ \operatorname{LR}_M=8. \]

A likelihood ratio:

greater than 1 favors spam;
equal to 1 provides no distinction;
less than 1 favors legitimate email.

The likelihood ratio measures evidential strength, but it is not itself the posterior probability. It must be combined with the prior.

Bayes’ Theorem in Odds Form

Probability \(p\) can be converted to odds:\[ \text{odds}=\frac{p}{1-p}. \]

The prior spam probability is:\[ P(S)=0.20. \]

Therefore, the prior odds are:\[ \frac{0.20}{0.80} = \frac{1}{4}. \]

The likelihood ratio is:\[ 8. \]

Bayes’ theorem in odds form is:\[ \boxed{ \text{Posterior odds} = \text{Prior odds} \times \text{Likelihood ratio} } \]

Therefore:\[ \text{Posterior odds} = \frac{1}{4}\times8 = 2. \]

Odds of 2 correspond to probability:\[ p=\frac{2}{1+2}=\frac{2}{3}. \]

Thus:\[ P(S\mid M)=66.7\%. \]

The odds form is useful when updating with several pieces of evidence.

The Base Rate Matters

The prior probability \(P(S)\) is also called the base rate.

Suppose the word frequencies remain the same but only 1% of emails are spam:\[ P(S)=0.01. \]

Then:\[ P(H)=0.99. \]

The posterior becomes:\[ P(S\mid M) = \frac{ (0.08)(0.01) }{ (0.08)(0.01)+(0.01)(0.99) }. \]

Therefore:\[ P(S\mid M) = \frac{0.0008}{0.0107} \approx0.0748. \]

The posterior is now only about 7.5%.

Spam base rate	\(P(S\mid M)\)
1%	7.5%
5%	29.6%
20%	66.7%
50%	88.9%
80%	97.0%

The feature’s evidence is unchanged, but its interpretation changes with the prevalence of spam.

Ignoring this prevalence is called base-rate neglect.

Absence of the Word Is Also Evidence

A classifier can also update using:\[ M^c=\text{the email does not contain “money”}. \]

The relevant probabilities are:\[ P(M^c\mid S)=1-0.08=0.92 \]

and:\[ P(M^c\mid H)=1-0.01=0.99. \]

Using Bayes’ theorem:\[ P(S\mid M^c) = \frac{ P(M^c\mid S)P(S) }{ P(M^c\mid S)P(S) + P(M^c\mid H)P(H) }. \]

Substituting:\[ P(S\mid M^c) = \frac{ (0.92)(0.20) }{ (0.92)(0.20)+(0.99)(0.80) }. \]

Therefore:\[ P(S\mid M^c) = \frac{0.184}{0.976} \approx0.1885. \]

Not observing the word reduces the spam probability from 20% to about 18.9%, but only slightly. The word’s presence is much stronger evidence than its absence.

General Bayes’ Theorem for Multiple Classes

Suppose \(C_1,C_2,\ldots,C_k\) form a partition of all possible classes.

After observing evidence \(E\):\[ \boxed{ P(C_i\mid E) = \frac{ P(E\mid C_i)P(C_i) }{ \sum_{j=1}^{k}P(E\mid C_j)P(C_j) } } \]

The denominator ensures that the posterior probabilities sum to one:\[ \sum_{i=1}^{k}P(C_i\mid E)=1. \]

A message classifier might use classes such as:

legitimate;
promotional;
phishing;
malware;
financial fraud.

Bayes’ theorem applies in the same way.

Multiple Words and Naïve Bayes

A realistic email filter considers many features:\[ X=(X_1,X_2,\ldots,X_d). \]

The classifier wants:\[ P(S\mid X). \]

Bayes’ theorem gives:\[ P(S\mid X) = \frac{ P(X\mid S)P(S) }{ P(X) }. \]

The difficult term is:\[ P(X\mid S), \]

because words and features can interact in many combinations.

Naïve Bayes simplifies the problem by assuming conditional independence:\[ P(X_1,\ldots,X_d\mid S) \approx \prod_{j=1}^{d}P(X_j\mid S). \]

Similarly:\[ P(X_1,\ldots,X_d\mid H) \approx \prod_{j=1}^{d}P(X_j\mid H). \]

The model can then compare:\[ P(S)\prod_jP(X_j\mid S) \]

with:\[ P(H)\prod_jP(X_j\mid H). \]

The conditional-independence assumption is often imperfect because words occur in phrases and topics. Nevertheless, the resulting classifier can still be useful.

Classification Requires a Decision Threshold

A posterior probability does not decide by itself how an email should be handled.

A simple rule might be:\[ \text{Classify as spam if } P(S\mid X)>0.5. \]

But 0.5 is not always the appropriate threshold.

The choice depends on the consequences of:

a legitimate email being blocked;
a spam email reaching the inbox.

If blocking an important legitimate email is especially costly, the system may require a much higher posterior probability before placing it in the spam folder.

Possible actions include:

deliver normally;
mark as suspicious;
place in a spam folder;
quarantine;
request user confirmation.

Probability estimation and decision-making are related but distinct steps.

Numerical Stability

When a model multiplies many probabilities:\[ \prod_{j=1}^{d}P(X_j\mid S), \]

the result may become extremely small.

Implementations therefore often work with logarithms:\[ \log \left[ P(S) \prod_jP(X_j\mid S) \right] = \log P(S) + \sum_j\log P(X_j\mid S). \]

This converts multiplication into addition and reduces numerical underflow.

The class with the highest log score is also the class with the highest original product because the logarithm is strictly increasing.

Smoothing Unseen Features

Suppose a word never appears in the spam training data. Its estimated probability might be zero:\[ P(X_j\mid S)=0. \]

Multiplying by zero would make the complete spam likelihood zero, regardless of every other feature.

Smoothing assigns small nonzero probabilities to unseen events. A common method is additive smoothing.

For word count \(c_{j,S}\), class token count \(N_S\), vocabulary size \(V\), and smoothing parameter \(\alpha>0\):\[ P(X_j\mid S) = \frac{ c_{j,S}+\alpha }{ N_S+\alpha V }. \]

This prevents one unseen feature from eliminating an entire class.

Bayes’ Theorem Does Not Guarantee a Good Model

Bayes’ theorem is mathematically exact, but its output is only as reliable as its inputs.

Potential problems include:

inaccurate class prevalence;
unrepresentative training data;
changing spam behavior;
poor feature definitions;
violated independence assumptions;
mislabeled examples;
deliberate adversarial wording;
population drift.

The theorem performs the update correctly given the specified probabilities. It does not guarantee that those probabilities represent the current environment accurately.

Common Mistakes

Reversing conditional probabilities

\[ P(M\mid S) \]

and:\[ P(S\mid M) \]

have different denominators and generally differ.

Omitting the prior

The posterior depends on the initial class prevalence:\[ P(S). \]

Using only one path in the denominator

The evidence can arise under every possible class:\[ P(M) = P(M\mid S)P(S) + P(M\mid H)P(H). \]

Treating the likelihood as the posterior