Likelihood Ratio (LR)

A likelihood ratio measures how much more likely the observed data is under one hypothesis than under another. It is the ratio of two likelihood values:

\[ \Lambda = \frac{L(\text{data} \mid H_1)}{L(\text{data} \mid H_0)} \]

A ratio of 1 means the data is equally likely under both. Above 1 the data favors \(H_1\); below 1 it favors \(H_0\). The reading is direct and it is entirely about the data, which is both the appeal and the limit.

The coin, and the two things called lambda

import numpy as np
from scipy.stats import binom, chi2, norm

n, k = 10, 7
L0 = binom.pmf(k, n, 0.5)
L1 = binom.pmf(k, n, 0.7)
print(f"  ten tosses, {k} heads. H0: p = 0.5, H1: p = 0.7")
print(f"    L(data | H0) = C(10,7) * 0.5^7 * 0.5^3 = {L0:.6f}")
print(f"    L(data | H1) = C(10,7) * 0.7^7 * 0.3^3 = {L1:.6f}")
print(f"    ratio        = {L1 / L0:.6f}")
print(f"  the binomial coefficient is in both lines and cancels:")
print(f"    (0.7/0.5)^7 * (0.3/0.5)^3 = {(0.7 / 0.5) ** 7 * (0.3 / 0.5) ** 3:.6f}")
print("  which is why the same ratio comes out of a design that stops after ten")
print("  tosses and one that stops after three tails")

phat = k / n
lam_simple = L1 / L0
lam_lrt = L0 / binom.pmf(k, n, phat)
stat = -2 * np.log(lam_lrt)
print(f"\n  two quantities share the symbol, and they point opposite ways:")
print(f"{'':>34}{'value':>12}{'large means':>16}")
print(f"{'L(H1)/L(H0), two simple points':>34}{lam_simple:>12.6f}{'favours H1':>16}")
print(f"{'sup L under H0 / sup L, all theta':>34}{lam_lrt:>12.6f}{'favours H0':>16}")
print(f"  the second uses the maximum-likelihood estimate {phat:.1f} in its denominator,")
print(f"  so it cannot exceed 1. Wilks turns it into a test:")
print(f"    -2 log(lambda) = {stat:.6f}, chi-square with 1 df -> p ="
      f" {chi2.sf(stat, 1):.6f}")
print(f"  an exact two-sided binomial test on the same data gives p ="
      f" {2 * min(binom.cdf(k, n, 0.5), binom.sf(k - 1, n, 0.5)):.6f},")
print("  which is nearly twice as large. The next table says why")

rng = np.random.default_rng(0)
runs = 200_000
print(f"\n  and the chi-square approximation has to be earned:")
print(f"{'tosses per experiment':>23}{'P(stat > 3.841)':>18}{'nominal':>10}"
      f"{'P(stat > 6.635)':>18}{'nominal':>10}")
for m in (10, 30, 100, 1000):
    x = binom.ppf(rng.random(runs), m, 0.5)
    p = x / m
    with np.errstate(divide="ignore", invalid="ignore"):
        ll1 = np.where(x > 0, x * np.log(np.where(p > 0, p, 1)), 0.0) \
            + np.where(x < m, (m - x) * np.log(np.where(p < 1, 1 - p, 1)), 0.0)
    ll0 = m * np.log(0.5)
    s = -2 * (ll0 - ll1)
    print(f"{m:>23,}{np.mean(s > chi2.ppf(0.95, 1)):>18.4f}{0.05:>10.4f}"
          f"{np.mean(s > chi2.ppf(0.99, 1)):>18.4f}{0.01:>10.4f}")
print("  at ten tosses the test rejects a true null 0.1084 of the time at a")
print("  nominal 0.05 - more than twice as often as advertised - and the 1%")
print("  level is off by a similar factor. Convergence is not monotone either,")
print("  because the outcome is discrete: thirty tosses under-reject. Wilks'")
print("  theorem is asymptotic, and 'large sample' is a claim to check")

print(f"\n  what a ratio of 10 does not mean:")
print(f"{'prior odds for H1':>20}{'likelihood ratio':>19}{'posterior odds':>17}"
      f"{'posterior P(H1)':>18}")
for prior in (1 / 9, 1 / 3, 1.0, 3.0):
    post = prior * 10
    print(f"{prior:>20.4f}{10:>19}{post:>17.4f}{post / (1 + post):>18.4f}")
print("  the ratio is what the data contributes and nothing else. Calling 10")
print("  'strong evidence' is a convention about that contribution, not a")
print("  statement that the hypothesis is probably true")
#   ten tosses, 7 heads. H0: p = 0.5, H1: p = 0.7
#     L(data | H0) = C(10,7) * 0.5^7 * 0.5^3 = 0.117187
#     L(data | H1) = C(10,7) * 0.7^7 * 0.3^3 = 0.266828
#     ratio        = 2.276932
#   the binomial coefficient is in both lines and cancels:
#     (0.7/0.5)^7 * (0.3/0.5)^3 = 2.276932
#   which is why the same ratio comes out of a design that stops after ten
#   tosses and one that stops after three tails
#
#   two quantities share the symbol, and they point opposite ways:
#                                          value     large means
#     L(H1)/L(H0), two simple points    2.276932      favours H1
#  sup L under H0 / sup L, all theta    0.439188      favours H0
#   the second uses the maximum-likelihood estimate 0.7 in its denominator,
#   so it cannot exceed 1. Wilks turns it into a test:
#     -2 log(lambda) = 1.645658, chi-square with 1 df -> p = 0.199551
#   an exact two-sided binomial test on the same data gives p = 0.343750,
#   which is nearly twice as large. The next table says why
#
#   and the chi-square approximation has to be earned:
#   tosses per experiment   P(stat > 3.841)   nominal   P(stat > 6.635)   nominal
#                      10            0.1084    0.0500            0.0217    0.0100
#                      30            0.0427    0.0500            0.0162    0.0100
#                     100            0.0573    0.0500            0.0121    0.0100
#                   1,000            0.0545    0.0500            0.0106    0.0100
#   at ten tosses the test rejects a true null 0.1084 of the time at a
#   nominal 0.05 - more than twice as often as advertised - and the 1%
#   level is off by a similar factor. Convergence is not monotone either,
#   because the outcome is discrete: thirty tosses under-reject. Wilks'
#   theorem is asymptotic, and 'large sample' is a claim to check
#
#   what a ratio of 10 does not mean:
#    prior odds for H1   likelihood ratio   posterior odds   posterior P(H1)
#               0.1111                 10           1.1111            0.5263
#               0.3333                 10           3.3333            0.7692
#               1.0000                 10          10.0000            0.9091
#               3.0000                 10          30.0000            0.9677
#   the ratio is what the data contributes and nothing else. Calling 10
#   'strong evidence' is a convention about that contribution, not a
#   statement that the hypothesis is probably true

What each block is for

The first works the example: ten tosses, seven heads, \(H_0: p = 0.5\) against \(H_1: p = 0.7\), giving likelihoods of 0.117187 and 0.266828 and a ratio of 2.276932. The data is about 2.3 times more likely under \(H_1\). The second line matters more than it looks — the binomial coefficient appears in both likelihoods and cancels, so the ratio depends only on the observed counts and not on the design that produced them. A ratio computed from a fixed number of tosses and one computed from a design that stopped at the third tail come out the same.

The second block separates two quantities that share a symbol and point in opposite directions. The simple ratio above compares two fully specified hypotheses and is large when \(H_1\) is favored. The likelihood ratio test statistic is

\[ \Lambda = \frac{\sup_{\theta \in \Theta_0} L(\theta)}{\sup_{\theta \in \Theta} L(\theta)} \]

whose denominator maximizes over the whole parameter space and therefore cannot be exceeded by the numerator. It lies in \([0, 1]\), and small values are evidence against \(H_0\). Under regularity conditions for identifiable nested models with the true null parameter in the interior, Wilks’ theorem gives an asymptotic chi-square distribution for \(-2 \log \Lambda\), with degrees of freedom equal to the difference in model dimensions. Boundary parameters, unidentified nuisance parameters, or other nonregular cases can require a different reference distribution. On the coin data that gives 1.645658 and \(p = 0.199551\).

The third block is why that p-value should not be taken at face value here. Under a true null, the statistic exceeds the 95th percentile of its chi-square reference 0.1084 of the time at ten tosses — more than twice the nominal rate — and convergence is not monotone, because the outcome is discrete. At a thousand observations the simulated 5% rejection rate is closer to nominal, at 0.0545; this does not certify approximation accuracy for every application. “Large sample” is a property to verify for the model at hand, not a phrase to wave through, and on small discrete data an exact test is the better tool.

The fourth block states the limit of the quantity itself. A ratio of 10 becomes a posterior probability only once prior odds are supplied: at 1:9 it produces a posterior probability of 0.5263, at even odds 0.9091. The conventional bands — around 1 is no evidence, above 3 moderate, above 10 strong — describe the data’s contribution, and they are a convention rather than a result.

Where it turns up

The likelihood ratio test is the standard tool for comparing nested models, where the null fixes some parameters and the alternative frees them. The sequential probability ratio test updates the ratio as data arrives, \(\Lambda_n\) computed on everything seen so far, and stops when it crosses an upper threshold \(A\) in favor of \(H_1\) or a lower threshold \(B\) in favor of \(H_0\), continuing otherwise. In Bayesian work, a Bayes factor is a likelihood ratio in which each hypothesis has been averaged over a prior rather than fixed at a point — which is why a Bayes factor depends on that prior and a simple ratio between two fully specified hypotheses does not. In diagnostics, the positive and negative likelihood ratios of a test, LR+ and LR−, combine with a patient’s pre-test odds in exactly the way the fourth block sets out.

The cancellation in the first block is what unites these. Because the ratio uses only the likelihood of what was observed, it carries no information about the stopping rule, which is what makes sequential use of it coherent and what separates it from a p-value computed over outcomes that did not occur.

References

• SciPy documentation, scipy.stats.binom and scipy.stats.chi2. docs.scipy.org