Conditional Probability and the Law of Total Probability
Conditional probability describes how the probability of an event changes after additional information becomes available.
An email containing the word “money” may be more likely to be spam than an email selected without examining its contents. Conversely, the probability that “money” appears may differ depending on whether the email is spam or legitimate.
These are different conditional probabilities:\[ P(\text{Money}\mid\text{Spam}) \]
and:\[ P(\text{Spam}\mid\text{Money}). \]
Understanding their difference is essential for probability, statistical inference, diagnostic testing, and classification systems.
Defining the Events
Let:\[ S=\text{the email is spam} \]
and:\[ H=\text{the email is legitimate}. \]
Legitimate email is sometimes called ham.
Assume every email belongs to exactly one of these two categories. Then:\[ H=S^c. \]
Therefore:\[ P(H)=1-P(S). \]
Also define:\[ M=\text{the email contains the word “money”}. \]
Suppose the following illustrative probabilities are given:\[ P(M\mid S)=0.08 \]
and:\[ P(M\mid H)=0.01. \]
These statements mean:
- 8% of spam emails contain the word;
- 1% of legitimate emails contain the word.
The values are conditional because each percentage is calculated within one email category.
What Does “Given” Mean?
The vertical bar in:\[ P(M\mid S) \]
is read as “given.”
Thus:\[ P(M\mid S) \]
means:
The probability that an email contains “money,” given that the email is known to be spam.
Conditioning restricts attention to the cases in which the condition is true.
Instead of examining all emails, \(P(M\mid S)\) examines only spam emails and asks what fraction contain the word.
Definition of Conditional Probability
For events \(A\) and \(B\), with:\[ P(A)>0, \]
the conditional probability of \(B\) given \(A\) is:\[ \boxed{ P(B\mid A) = \frac{P(A\cap B)}{P(A)} } \]
The numerator is the probability that both events occur:\[ P(A\cap B). \]
The denominator restricts the reference population to event \(A\).
In the email example:\[ P(M\mid S) = \frac{P(M\cap S)}{P(S)}. \]
This means:\[ \frac{ \text{probability that an email is spam and contains “money”} }{ \text{probability that an email is spam} }. \]
Conditional Probability as a Restricted Sample Space
Suppose 10,000 emails contain:
- 2,000 spam emails;
- 8,000 legitimate emails.
If 8% of spam emails contain “money,” then:\[ (0.08)(2000)=160 \]
spam emails contain it.
Within the spam group:\[ P(M\mid S) = \frac{160}{2000} = 0.08. \]
The denominator is 2,000 rather than 10,000 because the condition \(S\) restricts the reference set to spam emails.
Similarly, if 1% of legitimate emails contain the word:\[ (0.01)(8000)=80. \]
Therefore:\[ P(M\mid H) = \frac{80}{8000} = 0.01. \]
The General Multiplication Rule
Starting with:\[ P(B\mid A) = \frac{P(A\cap B)}{P(A)}, \]
multiply both sides by \(P(A)\):\[ \boxed{ P(A\cap B) = P(A)P(B\mid A) } \]
This is the general multiplication rule.
The order can also be reversed:\[ \boxed{ P(A\cap B) = P(B)P(A\mid B) } \]
Therefore:\[ P(A)P(B\mid A) = P(B)P(A\mid B). \]
This identity forms the foundation of Bayes’ theorem.
Calculating Joint Email Probabilities
Suppose:\[ P(S)=0.20. \]
Then:\[ P(H)=1-0.20=0.80. \]
The probability that an email is both spam and contains “money” is:\[ P(M\cap S) = P(S)P(M\mid S). \]
Substituting the values:\[ P(M\cap S) = (0.20)(0.08) = 0.016. \]
Thus:\[ P(M\cap S)=1.6\%. \]
Likewise:\[ P(M\cap H) = P(H)P(M\mid H) \]
so:\[ P(M\cap H) = (0.80)(0.01) = 0.008. \]
Therefore:\[ P(M\cap H)=0.8\%. \]
Partitioning the Event
Every email containing “money” must belong to one of two mutually exclusive groups:
- it contains “money” and is spam;
- it contains “money” and is legitimate.
Therefore:\[ M = (M\cap S)\cup(M\cap H). \]
Because an email cannot be both spam and legitimate:\[ (M\cap S)\cap(M\cap H)=\varnothing. \]
The addition rule gives:\[ P(M) = P(M\cap S)+P(M\cap H). \]
Substituting the joint probabilities:\[ P(M) = 0.016+0.008 = 0.024. \]
Thus:\[ \boxed{ P(M)=2.4\% } \]
A randomly selected email has a 2.4% probability of containing the word under this illustrative model.
The Law of Total Probability
The previous calculation is an example of the law of total probability.
If events \(A_1,A_2,\ldots,A_k\) form a partition of the sample space, then:
- they are mutually exclusive;
- one of them must occur;
- together they cover the entire sample space.
For any event \(B\):\[ \boxed{ P(B) = \sum_{i=1}^{k} P(B\mid A_i)P(A_i) } \]
For spam and legitimate email:\[ P(M) = P(M\mid S)P(S) + P(M\mid H)P(H). \]
Numerically:\[ P(M) = (0.08)(0.20) + (0.01)(0.80) = 0.024. \]
The law calculates an overall probability by averaging conditional probabilities according to the prevalence of their groups.
Weighted-Average Interpretation
The total probability:\[ P(M)=0.024 \]
is a weighted average of:\[ P(M\mid S)=0.08 \]
and:\[ P(M\mid H)=0.01. \]
The weights are:\[ P(S)=0.20 \]
and:\[ P(H)=0.80. \]
Therefore:\[ P(M) = (0.20)(0.08) + (0.80)(0.01). \]
Because legitimate emails are more common in this example, their 1% conditional rate receives more weight.
As with any weighted average:\[ 0.01\leq0.024\leq0.08. \]
The overall probability must lie between the two group-specific probabilities.
Frequency Table
The same calculation can be displayed using 10,000 hypothetical emails.
| Email type | Contains “money” | Does not contain “money” | Total |
|---|---|---|---|
| Spam | 160 | 1,840 | 2,000 |
| Legitimate | 80 | 7,920 | 8,000 |
| Total | 240 | 9,760 | 10,000 |
From the table:\[ P(M\mid S) = \frac{160}{2000} = 0.08. \]\[ P(M\mid H) = \frac{80}{8000} = 0.01. \]\[ P(M) = \frac{240}{10000} = 0.024. \]
Frequency tables often make conditional probabilities easier to understand because they make the changing denominator visible.
Probability Tree
A tree diagram provides another representation:
Email├── Spam: 0.20│ ├── Money: 0.08 → 0.20 × 0.08 = 0.016│ └── No money: 0.92 → 0.20 × 0.92 = 0.184└── Legitimate: 0.80 ├── Money: 0.01 → 0.80 × 0.01 = 0.008 └── No money: 0.99 → 0.80 × 0.99 = 0.792
Multiply probabilities along a branch:\[ P(S\cap M) = P(S)P(M\mid S). \]
Add mutually exclusive branches to calculate an overall event:\[ P(M) = P(S\cap M)+P(H\cap M). \]
The leaf probabilities sum to 1:\[ 0.016+0.184+0.008+0.792=1. \]
Independence as a Special Case
Events \(A\) and \(B\) are independent if knowing that \(A\) occurred does not change the probability of \(B\):\[ P(B\mid A)=P(B). \]
Substituting this into the general multiplication rule:\[ P(A\cap B) = P(A)P(B\mid A) \]
gives:\[ \boxed{ P(A\cap B) = P(A)P(B) } \]
Thus, the familiar independent multiplication rule is a special case of the conditional multiplication rule.
In the email example:\[ P(M\mid S)=0.08 \]
while:\[ P(M)=0.024. \]
Because these probabilities differ:\[ P(M\mid S)\neq P(M), \]
the events \(M\) and \(S\) are not independent under this model.
The occurrence of the word provides information about spam status.
Reversing the Conditional Probability
The calculation:\[ P(M\mid S)=0.08 \]
does not mean:\[ P(S\mid M)=0.08. \]
These probabilities answer different questions.
Word given spam
\[ P(M\mid S) \]
asks:
Among spam emails, what proportion contains the word?
Spam given word
\[ P(S\mid M) \]
asks:
Among emails containing the word, what proportion is spam?
Using the table:\[ P(S\mid M) = \frac{160}{240} = \frac{2}{3} \approx0.667. \]
Thus, under the illustrative values:\[ \boxed{ P(S\mid M)\approx66.7\% } \]
Even though only 8% of spam contains the word, about two-thirds of emails containing it are spam. The difference arises because the two calculations use different denominators.
Bayes’ Theorem
From the general multiplication rule:\[ P(A\cap B) = P(A)P(B\mid A) \]
and:\[ P(A\cap B) = P(B)P(A\mid B), \]
we obtain:\[ P(A)P(B\mid A) = P(B)P(A\mid B). \]
Solving for \(P(A\mid B)\):\[ \boxed{ P(A\mid B) = \frac{ P(B\mid A)P(A) }{ P(B) } } \]
For the email example:\[ P(S\mid M) = \frac{ P(M\mid S)P(S) }{ P(M) }. \]
Substitute the values:\[ P(S\mid M) = \frac{ (0.08)(0.20) }{ 0.024 } = \frac{0.016}{0.024} = \frac{2}{3}. \]
Bayes’ theorem reverses the direction of conditioning.
Prior and Posterior Probability
In a classification context:\[ P(S)=0.20 \]
is the prior probability of spam before examining the word.
After observing “money,” the posterior probability is:\[ P(S\mid M)\approx0.667. \]
The evidence changes the probability from:\[ 20\% \]
to approximately:\[ 66.7\%. \]
This does not prove that the message is spam. It updates the degree of uncertainty based on the observed feature.
Why the Base Rate Matters
Suppose the word frequencies remain:\[ P(M\mid S)=0.08 \]
and:\[ P(M\mid H)=0.01, \]
but spam becomes rare:\[ P(S)=0.01. \]
Then:\[ P(H)=0.99. \]
The overall probability of the word is:\[ P(M) = (0.08)(0.01) + (0.01)(0.99) \]\[ = 0.0008+0.0099 = 0.0107. \]
The posterior spam probability becomes:\[ P(S\mid M) = \frac{0.0008}{0.0107} \approx0.0748. \]
Thus, only about 7.5% of emails containing the word would be spam.
The same feature can imply very different posterior probabilities depending on the base rate:\[ P(S). \]
Ignoring the base rate is known as the base-rate fallacy.
Information Provided by a Feature
The word is informative when:\[ P(M\mid S) \]
differs from:\[ P(M\mid H). \]
A useful comparison is the likelihood ratio:\[ \operatorname{LR} = \frac{ P(M\mid S) }{ P(M\mid H) }. \]
For the illustrative values:\[ \operatorname{LR} = \frac{0.08}{0.01} = 8. \]
The word is eight times as likely to appear in spam as in legitimate email.
However, the likelihood ratio alone is not the posterior probability. It must be combined with the prior prevalence.
Multiple Features in Spam Filtering
A realistic spam filter considers many features:
- words and phrases;
- sender reputation;
- links;
- formatting;
- attachment types;
- message structure;
- user feedback.
A simplified model may estimate:\[ P(S\mid X_1,X_2,\ldots,X_k), \]
where \(X_j\) represents the presence or absence of feature \(j\).
A naïve Bayes classifier assumes that features are conditionally independent given the email class:\[ P(X_1,\ldots,X_k\mid S) \approx \prod_{j=1}^{k}P(X_j\mid S). \]
It similarly calculates:\[ P(X_1,\ldots,X_k\mid H) \approx \prod_{j=1}^{k}P(X_j\mid H). \]
This assumption is often unrealistic because words can be related, but the method can still provide a useful classifier in some settings.
One word alone should not be treated as definitive evidence. Classification should combine multiple signals and consider the consequences of false positives and false negatives.
General Total Probability with Several Groups
Suppose emails are divided into three categories:\[ C_1=\text{promotional spam} \]\[ C_2=\text{fraudulent spam} \]\[ C_3=\text{legitimate email}. \]
If these categories form a partition, then:\[ P(M) = P(M\mid C_1)P(C_1) + P(M\mid C_2)P(C_2) + P(M\mid C_3)P(C_3). \]
The law of total probability works with any finite or countable partition, not only a binary split.
Common Mistakes
Reversing the condition
\[ P(M\mid S) \neq P(S\mid M) \]
in general.
Forgetting the base rate
A feature may be common among spam but still produce a low spam probability if spam itself is rare.
Adding conditional probabilities directly
This is incorrect:\[ P(M) \neq P(M\mid S)+P(M\mid H). \]
The conditional probabilities must be weighted:\[ P(M) = P(M\mid S)P(S) + P(M\mid H)P(H). \]
Assuming dependence implies causation
The word and spam status are associated, but the probability calculation alone does not establish a causal claim.
Treating a posterior probability as certainty
A 66.7% posterior probability still leaves a 33.3% probability of the alternative under the model.
Assuming all features are independent
Words and message characteristics can be related. Independence assumptions should be understood as modeling approximations.
A Reliable Calculation Process
When solving a conditional-probability problem:
- Define the events clearly.
- Identify which probability is requested.
- Pay attention to the order around the conditioning bar.
- Write the available conditional and prior probabilities.
- Partition the sample space into mutually exclusive cases.
- Multiply along branches.
- Add across mutually exclusive branches.
- Use Bayes’ theorem when reversing the condition.
- Check that the result lies between 0 and 1.
- Interpret the result using the correct reference group.
Key Takeaway
Conditional probability changes the reference population to cases satisfying the given condition. The general multiplication rule,\[ P(A\cap B)=P(A)P(B\mid A), \]
calculates joint probabilities. The law of total probability combines mutually exclusive cases:\[ P(B)=\sum_iP(B\mid A_i)P(A_i). \]
In the illustrative spam example, the overall probability that an email contains “money” is 2.4%. Bayes’ theorem then reverses the condition and gives a spam probability of approximately 66.7% among emails containing that word. The calculation depends not only on the feature’s frequency but also on the underlying spam base rate.