Bayes’ Theorem

Bayes' theorem says how a probability should change when evidence arrives. For a hypothesis \(H\) and data \(D\):

\[ P(H \mid D) = \frac{P(D \mid H)\,P(H)}{P(D)} \]

The prior \(P(H)\) is the belief before the data, the likelihood \(P(D \mid H)\) is how well the hypothesis accounts for what was seen, and \(P(D)\) normalizes so the result is a probability. What comes out is the posterior, \(P(H \mid D)\).

The theorem itself is not controversial — it follows in one line from the definition of conditional probability. What makes it worth a glossary entry is that people who know it still get its consequences wrong, in one predictable way.

Why the denominator matters

The joint probability of H and D can be written either as P(D | H)P(H) or as P(H | D)P(D). Equating them and dividing by P(D), when P(D) is positive, gives the theorem. For an exhaustive set of mutually exclusive hypotheses, the denominator is the sum of each hypothesis’s prior probability times its likelihood for D. It includes all modeled ways the evidence could occur, not just the explanation being investigated.

Consider a hypothetical manufacturing inspection. One percent of items are defective, the inspection flags 99% of defective items, and it also flags 5% of nondefective items. In a population of 10,000 items, the expected counts are 100 defective items with 99 flagged, and 9,900 nondefective items with 495 flagged. Among 594 flagged items, 99 are defective: 99/594 = 1/6, about 16.7%. These counts express assumed rates, not measured performance of a real inspection system.

The numerator is P(flagged | defective)P(defective) = 0.99 × 0.01 = 0.0099. The denominator adds flags from nondefective items: 0.0099 + 0.05 × 0.99 = 0.0594. Sensitivity, the probability of a flag given a defect, answers a different question from the probability of a defect given a flag. Swapping these conditional probabilities ignores the much larger nondefective population.

If half the items were defective and the same conditional inspection rates remained valid, the fraction of flagged items that were defective would instead be 0.99 × 0.5 / (0.99 × 0.5 + 0.05 × 0.5), about 95.2%. In an application, changing the population can also change the conditional rates. Transporting this calculation therefore requires evidence for both the base rate and the inspection’s behavior in the target population.

Odds make the structure visible

Odds are p/(1 − p), the probability of an event divided by that of its complement. In this example, prior odds of a defect are 1/99. The positive likelihood ratio is 0.99/0.05 = 19.8, giving posterior odds of 19.8/99 = 0.2, or 1 to 5. Convert odds o back to probability using o/(1 + o): 0.2/1.2 = 1/6. The multiplier remains 19.8 only while those conditional inspection rates stay fixed.

\[ \text{posterior odds} = \text{prior odds} \times \frac{P(D \mid H)}{P(D \mid \neg H)} \]

That form names the error precisely. Under these fixed conditional rates, the evidence provides a fixed odds multiplier; the conclusion is that multiplier applied to a base rate. Base rate neglect is reading the 19.8 and forgetting the number it multiplies. It also explains why the same evidence can be decisive in one setting and insufficient for the same decision in another without anything about the evidence having changed.

In this numerical example, false positives outnumber true positives because the base rate and false-positive rate combine that way. A rare condition alone does not imply low positive predictive value. For a second result \(D_2\), multiply the current odds by \(P(D_2\mid H,D_1)/P(D_2\mid\neg H,D_1)\). Only if the two results are conditionally independent given each hypothesis, and the second test has the same likelihood ratio, does the multiplier become another 19.8. Repeating the very same result supplies no new evidence.

What it does and does not settle

Bayes' theorem is a rule for updating, not a source of beliefs. It takes a prior and returns a posterior; it has nothing to say about where the prior came from. In the numerical example the defect rate was specified as an input; an application would need to justify its relevance to the inspected population. When the prior encodes a judgment instead, the theorem propagates that judgment faithfully into the answer — see prior belief for what that commits you to.

Updating is conditional on the specified model. In a fixed discrete hypothesis space, a hypothesis given prior mass zero remains at zero after ordinary conditioning on evidence with positive marginal probability. Evidence can reveal that the model is inadequate, but revising its hypothesis space is a separate modeling step.

Used well, it underlies posterior probabilities in experiments, naive Bayes classifiers, Bayesian neural networks and diagnostic reasoning. In every case the machinery is this one line, and the difficulty is never the line.

References

• SciPy documentation, scipy.stats.binom. docs.scipy.org
• NumPy documentation, mathematical functions. numpy.org