1. Definition

  • The posterior probability is the updated probability of a hypothesis (or parameter value) after observing data, using Bayes’ theorem.
  • It combines:
    1. Prior probability (belief before seeing data), and
    2. Likelihood (evidence from data).

Formula:

$P(H \mid \text{data}) = \frac{P(\text{data} \mid H) \cdot P(H)}{P(\text{data})}$

Where:

  • $P(H)$ = prior probability of hypothesis H
  • $P(\text{data} \mid H)$ = likelihood of the observed data given H
  • $P(\text{data})$ = marginal probability of data (normalizing constant)

2. Interpretation

  • Posterior probability answers:
    “Given the data I’ve observed, how probable is this hypothesis (or parameter value)?”
  • Very different from frequentist p-values, which answer:
    “If H₀ were true, how likely is the data (or more extreme)?”

3. Example – Coin Toss

Suppose we want to infer if a coin is biased toward heads.

  • Hypotheses:
    • $H_0$: coin is fair (p = 0.5)
    • $H_1$​: coin is biased (p = 0.7)
  • Prior: $P(H_0) = P(H_1) = 0.5$ (equal belief).
  • Data: 10 tosses, 7 heads.

Likelihoods:

  • $P(\text{data} \mid H_0) = {10 \choose 7} (0.5)^{10} ≈ 0.117$
  • $P(\text{data} \mid H_1) = {10 \choose 7} (0.7)^7 (0.3)^3 ≈ 0.266$

Posterior for H₁:

$P(H_1 \mid \text{data}) = \frac{0.266 \cdot 0.5}{0.266 \cdot 0.5 + 0.117 \cdot 0.5} = \frac{0.133}{0.133 + 0.0585} \approx 0.694$

Interpretation: After seeing 7 heads in 10 tosses, the probability that the coin is biased is about 69%.

4. Posterior Distribution (Parameters)

  • If we estimate parameters (not just discrete hypotheses), the posterior becomes a distribution:

$P(\theta \mid \text{data}) \propto P(\text{data} \mid \theta) \cdot P(\theta)$

Where:

  • $P(\theta)$ = prior distribution
  • $P(\text{data} \mid \theta)$ = likelihood
  • $P(\theta \mid \text{data})$ = posterior distribution

This is used for Bayesian inference, e.g., estimating means, regression coefficients, conversion rates.

5. Applications

  • Bayesian sequential testing: stop experiment when posterior $P(H_1 \mid data)$ > 0.95.
  • Bayesian estimation: use posterior distribution instead of point estimates.
  • Machine learning: Bayesian models (Naive Bayes, Bayesian neural nets).
  • Medical decisions: posterior probability of disease given test result.

6. Key Takeaways

  • Posterior probability = probability of a hypothesis given observed data.
  • Formula: Prior × Likelihood → normalized.
  • Provides a direct, intuitive interpretation (unlike p-values).
  • In Bayesian inference, all decisions are based on the posterior.

In short:
The posterior probability is the updated probability of a hypothesis or parameter after seeing data. It’s calculated using Bayes’ theorem and combines prior belief with evidence from data.