Likelihood
Definition
Likelihood measures how plausible a statistical model (with specific parameter values) is, given the observed data.
- In simple terms: “Given the data we observed, how likely is it that these parameter values generated it?”
- Contrast with probability:
- Probability: Given parameters, what’s the chance of seeing this data?
- Likelihood: Given data, how plausible are these parameters?
Formula
Suppose we have data points $x_1, x_2, …, x_n$, assumed independent and identically distributed (i.i.d.) from a distribution with parameter $\theta$.
- Probability of data:
$P(X = x \mid \theta)$
- Likelihood function:
$L(\theta \mid x) = \prod_{i=1}^n f(x_i \mid \theta)$
where $f(x_i \mid \theta)$ is the probability density (continuous) or probability mass (discrete).
Key Points
- Not a probability of parameters – Likelihood is not normalized across $\theta$.
- Relative measure – It allows us to compare which parameter values make the observed data more plausible.
- Used in estimation – The foundation of Maximum Likelihood Estimation (MLE), where we choose the parameter $\hat{\theta}$ that maximizes $L(\theta \mid x)$.
Example
Imagine flipping a coin 10 times, and you observe 7 heads and 3 tails.
- Let parameter $\theta = P(\text{head})$.
- The likelihood of seeing this data is:
$L(\theta) = \theta^7 (1-\theta)^3$
- If $\theta = 0.5$, then $L(0.5) = 0.5^7 \cdot 0.5^3 = 0.5^{10}$.
- If $\theta = 0.7$, then $L(0.7) = 0.7^7 \cdot 0.3^3$, which is larger.
- Thus, the data supports $\theta = 0.7$ more than $\theta = 0.5$.
Applications
- Parameter estimation: MLE is used in regression, classification, time series, etc.
- Bayesian statistics: Likelihood combines with prior → posterior distribution.
- Hypothesis testing: Likelihood ratio tests.
- Machine learning: Loss functions often come from likelihood (e.g., cross-entropy from Bernoulli likelihood).
In short:
Likelihood is a function of parameters given data. It tells us which parameter values are more consistent with the observed evidence.