Definition

Bayesian inference is a method of statistical inference where you update your belief about unknown parameters using Bayes’ Theorem:

$P(\theta \mid D) = \frac{P(D \mid \theta) \, P(\theta)}{P(D)}$

Where:

• $\theta$ = parameter(s) of interest (e.g., conversion rate, model weights)
• $D$ = observed data
• $P(\theta)$ = prior belief about parameters
• $P(D \mid \theta)$ = likelihood of data given parameters
• $P(\theta \mid D)$ = posterior (updated belief after seeing data)
• $P(D)$ = normalizing constant (marginal likelihood)

In plain words:

Prior belief + Evidence (data) → Posterior belief

---

Why It Matters

• Provides a principled way to update beliefs as new data arrives.
• Handles uncertainty explicitly → outputs distributions, not just point estimates.
• Flexible: naturally incorporates prior knowledge, small data, hierarchical structures.

---

Example 1 — Coin Flip

Suppose you want to estimate the probability a coin lands heads ($\theta$).

1. Prior: Before flipping, assume uniform prior: $\theta \sim \text{Beta}(1,1)$.
2. Data: You flip 10 times, observe 7 heads.
3. Likelihood: $P(D \mid \theta) = \theta^7 (1-\theta)^3$.
4. Posterior: $\theta \mid D \sim \text{Beta}(8,4)$.

The posterior mean = $8/(8+4) = 0.667$.
So after 10 flips, you believe the coin has ~67% chance of landing heads.

---

Example 2 — A/B Testing

• Prior: Conversion rate of Variant B is similar to A (Beta(1,1)).
• Data: A = 100/1000 convert (10%), B = 120/1000 convert (12%).
• Posterior: Update priors with Binomial likelihood → Beta distributions.
• Compare posteriors: Compute $P(\text{B > A})$.

Instead of just a p-value, you get:

“There’s a 94% probability that Variant B’s conversion rate is higher than A’s.”

Much easier to interpret for business stakeholders.

---

Frequentist vs Bayesian (Intuition)

Aspect	Frequentist	Bayesian
Parameters	Fixed but unknown	Random variables with distributions
Output	Point estimate + CI	Posterior distribution
Prior knowledge	Not used	Explicitly included
P-value	“How extreme is data under $H_0$​?”	“How probable is hypothesis given data?”
Interpretation	More rigid	More intuitive

---

Advantages

• Naturally incorporates prior knowledge.
• Provides probabilities of hypotheses (intuitive).
• Works well in sequential settings (online learning, adaptive A/B tests).
• Robust to small sample sizes.

---

Limitations

• Requires choosing a prior (can be subjective).
• Computationally expensive (MCMC, variational inference).
• Results depend on prior choice when data is sparse.

---

Applications

• Machine learning: Bayesian neural nets, Gaussian processes.
• A/B testing: posterior probability of uplift.
• Forecasting: Bayesian time series.
• Decision making: Bayesian decision theory, Thompson sampling in bandits.

---

Summary
Bayesian inference = continuously updating probability distributions for unknown parameters as new data arrives, using Bayes’ Theorem.
It’s powerful for uncertainty quantification, adaptive experimentation, and decision making.