The Law of Large Numbers and the Stability of Sample Averages
The law of large numbers explains why averages and proportions become increasingly stable as the sample size grows. It provides the mathematical foundation for estimating population characteristics through repeated observations or random sampling.
The Square-Root Law
Suppose \(X_1,X_2,\ldots,X_n\) are independent and identically distributed observations with population mean\[ E(X_i)=\mu \]
and finite standard deviation\[ SD(X_i)=\sigma. \]
The sample mean is\[ \bar{X}_n = \frac{X_1+X_2+\cdots+X_n}{n}. \]
Its expected value is\[ E(\bar{X}_n)=\mu, \]
and its standard error is\[ SE(\bar{X}_n) = \frac{\sigma}{\sqrt{n}}. \]
As \(n\) increases, the denominator \(\sqrt{n}\) increases, so the standard error decreases.
For example:
| Sample size \(n\) | Standard error |
|---|---|
| 25 | \(\sigma/5\) |
| 100 | \(\sigma/10\) |
| 400 | \(\sigma/20\) |
| 10,000 | \(\sigma/100\) |
Multiplying the sample size by four divides the standard error by two. Multiplying it by 100 divides the standard error by 10.
What the Law of Large Numbers Says
The law of large numbers states that, under suitable conditions, the sample mean approaches the population mean as the number of observations increases:\[ \bar{X}_n \longrightarrow \mu. \]
A more precise probability statement is\[ P\left( \left|\bar{X}_n-\mu\right|>\varepsilon \right) \longrightarrow 0 \qquad\text{as }n\to\infty \]
for every \(\varepsilon>0\).
This means that the probability of the sample mean differing from \(\mu\) by more than any fixed positive amount becomes arbitrarily small as the sample size increases.
The law does not say that a sample mean must become steadily closer to \(\mu\) after every additional observation. It says that large deviations become increasingly unlikely as the sample size grows.
Weak and Strong Laws
Two common versions of the law describe different forms of convergence.
Weak Law of Large Numbers
The weak law states that\[ \bar{X}_n\xrightarrow{P}\mu, \]
meaning that the sample mean converges to \(\mu\) in probability.
For every \(\varepsilon>0\),\[ P\left( \left|\bar{X}_n-\mu\right|>\varepsilon \right)\to0. \]
Strong Law of Large Numbers
The strong law states that\[ \bar{X}_n\xrightarrow{\text{a.s.}}\mu, \]
meaning that the sample mean converges to \(\mu\) almost surely.
Informally, if an infinite sequence of observations were generated, then—with probability 1—the running average would eventually approach and remain arbitrarily close to \(\mu\).
The strong law makes a stronger mathematical claim, but both versions express the same practical principle: averages stabilize around their expected value.
Why the Square-Root Law Supports the Result
If the observations have finite variance, then\[ \operatorname{Var}(\bar{X}_n) = \frac{\sigma^2}{n}. \]
Chebyshev’s inequality states that\[ P\left( \left|\bar{X}_n-\mu\right|\geq\varepsilon \right) \leq \frac{\sigma^2}{n\varepsilon^2}. \]
As \(n\to\infty\),\[ \frac{\sigma^2}{n\varepsilon^2}\to0. \]
Therefore,\[ P\left( \left|\bar{X}_n-\mu\right|\geq\varepsilon \right)\to0. \]
This proves a common finite-variance version of the weak law.
Example: Repeated Coin Tosses
Suppose a fair coin is tossed repeatedly. Define\[ X_i= \begin{cases} 1, & \text{if toss }i\text{ is a tail},\\ 0, & \text{if toss }i\text{ is a head}. \end{cases} \]
Since the coin is fair,\[ E(X_i)=0.5. \]
The sample mean is\[ \bar{X}_n = \frac{X_1+\cdots+X_n}{n}. \]
Because each tail contributes 1 and each head contributes 0, this sample mean is the proportion of tails:\[ \bar{X}_n = \frac{\text{number of tails}}{n}. \]
The law of large numbers says\[ \bar{X}_n\longrightarrow0.5. \]
After 10 tosses, the observed proportion might be 0.70. After 100 tosses, it might be 0.53. After 10,000 tosses, it might be 0.498.
The proportion is not guaranteed to improve monotonically, but substantial deviations from 0.5 become less likely with larger \(n\).
Percentages Are Averages
Suppose \(X_i\) indicates whether observation \(i\) belongs to a category:\[ X_i= \begin{cases} 1, & \text{if the observation belongs to the category},\\ 0, & \text{otherwise}. \end{cases} \]
If the true population proportion is \(p\), then\[ E(X_i)=p. \]
The sample proportion is\[ \hat{p} = \frac{X_1+\cdots+X_n}{n}. \]
Therefore, the law of large numbers gives\[ \hat{p}\longrightarrow p. \]
The sample percentage is \(100\hat{p}\), so\[ 100\hat{p}\longrightarrow100p. \]
This is why percentages calculated from large random samples tend to be more stable than percentages calculated from small samples.
What Happens to Sums?
Let\[ S_n=X_1+X_2+\cdots+X_n. \]
The expected value of the sum is\[ E(S_n)=n\mu, \]
and its standard error is\[ SE(S_n)=\sigma\sqrt{n}. \]
The absolute uncertainty of the sum increases with \(n\). Therefore, the unnormalized difference\[ S_n-n\mu \]
does not generally converge to zero.
However, it is too strong to say that the law of large numbers has no relevance to sums. The sample mean is the normalized sum:\[ \frac{S_n}{n}=\bar{X}_n. \]
The law states that\[ \frac{S_n}{n}\longrightarrow\mu. \]
Equivalently,\[ \frac{S_n-n\mu}{n}\longrightarrow0. \]
Thus, although the absolute error in the sum may increase, its error relative to the number of observations decreases.
Absolute Versus Relative Error
The standard error of the sum is\[ SE(S_n)=\sigma\sqrt{n}, \]
but the expected scale of the sum is proportional to \(n\):\[ E(S_n)=n\mu. \]
When \(\mu\neq0\), their ratio is\[ \frac{SE(S_n)}{|E(S_n)|} = \frac{\sigma\sqrt{n}}{n|\mu|} = \frac{\sigma}{|\mu|\sqrt{n}}. \]
This relative uncertainty approaches zero.
A sum becomes larger and may have greater absolute variability, but its average contribution per observation becomes increasingly stable.
A Coin-Tossing Illustration
Let \(S_n\) be the number of tails in \(n\) fair coin tosses.
Then\[ E(S_n)=\frac{n}{2} \]
and\[ SE(S_n)=\frac{\sqrt{n}}{2}. \]
The standard error increases as \(n\) grows:
| Number of tosses | Expected tails | Standard error |
|---|---|---|
| 100 | 50 | 5 |
| 10,000 | 5,000 | 50 |
| 1,000,000 | 500,000 | 500 |
The absolute standard error grows from 5 to 500. But as a proportion of \(n\), the standard error is\[ SE\left(\frac{S_n}{n}\right) = \frac{1}{2\sqrt{n}}. \]
This decreases:
| Number of tosses | Expected proportion | Standard error of proportion |
|---|---|---|
| 100 | 0.50 | 0.05 |
| 10,000 | 0.50 | 0.005 |
| 1,000,000 | 0.50 | 0.0005 |
The count becomes more variable in absolute terms, while the proportion becomes more precise.
Probability Distribution and Empirical Distribution
The law of large numbers also helps explain why an empirical distribution resembles its underlying probability distribution when the sample is large.
Suppose a random variable has possible outcomes \(a_1,a_2,\ldots,a_K\), with probabilities\[ P(X=a_j)=p_j. \]
For a particular outcome \(a_j\), define the indicator\[ I_i^{(j)} = \begin{cases} 1, & \text{if observation }i\text{ equals }a_j,\\ 0, & \text{otherwise}. \end{cases} \]
The empirical relative frequency of \(a_j\) is\[ \hat{p}_j = \frac{1}{n}\sum_{i=1}^{n}I_i^{(j)}. \]
Because\[ E\left(I_i^{(j)}\right)=p_j, \]
the law of large numbers implies\[ \hat{p}_j\longrightarrow p_j. \]
Thus, for each category, the observed relative frequency approaches its theoretical probability.
Example: Empirical Coin Distribution
For a fair coin, the theoretical probability distribution is
| Outcome | Theoretical probability |
|---|---|
| Heads | 0.50 |
| Tails | 0.50 |
A small empirical sample might produce:
| Outcome | Relative frequency after 10 tosses |
|---|---|
| Heads | 0.30 |
| Tails | 0.70 |
A larger sample might produce:
| Outcome | Relative frequency after 10,000 tosses |
|---|---|
| Heads | 0.499 |
| Tails | 0.501 |
The empirical distribution approaches the theoretical probability distribution as the number of tosses grows.
This convergence is not restricted to two-category outcomes. It applies to any finite or countable set of categories under appropriate assumptions.
Convergence of an Entire Empirical Distribution
For numerical data, define the population cumulative distribution function as\[ F(x)=P(X\leq x). \]
The empirical cumulative distribution function is\[ F_n(x) = \frac{1}{n} \sum_{i=1}^{n} \mathbf{1}(X_i\leq x). \]
For any fixed \(x\), the quantity \(F_n(x)\) is an average of 0–1 indicators. The law of large numbers therefore gives\[ F_n(x)\longrightarrow F(x). \]
A stronger result, known as the Glivenko–Cantelli theorem, states that under independent and identically distributed sampling,\[ \sup_x|F_n(x)-F(x)| \longrightarrow0 \quad\text{almost surely}. \]
This means the entire empirical cumulative distribution approaches the theoretical cumulative distribution uniformly, not merely at one selected point.
Sampling With and Without Replacement
Sampling with replacement produces independent observations when every draw uses the same population distribution. Independent computer simulations from a fixed probability model have the same structure.
However, the law of large numbers is not limited exclusively to literal sampling with replacement.
Independent Sampling
The standard version applies naturally to independent, identically distributed observations.
Sampling Without Replacement
If a sample is drawn without replacement from a fixed finite population, the observations are dependent. Moreover, the sample size cannot increase beyond the population size.
Nevertheless, the sample mean remains an unbiased estimator of the population mean under simple random sampling. If a sequence of increasingly large populations and samples is considered under suitable conditions, related consistency results can still hold.
When the sampling fraction is small, sampling without replacement behaves similarly to independent sampling. When it is large, the finite population correction reduces the standard error:\[ SE(\bar{X}) = \frac{\sigma}{\sqrt{n}} \sqrt{\frac{N-n}{N-1}}. \]
If the entire finite population is observed, \(n=N\), the sample mean equals the population mean exactly.
Conditions and Limitations
The law of large numbers depends on assumptions about the data-generating process.
Independence or Controlled Dependence
The simplest form assumes independent observations. Some dependent sequences also satisfy versions of the law, but arbitrary dependence can prevent averages from stabilizing.
Identical Distribution
The standard theorem assumes observations come from the same distribution. More general versions allow non-identical distributions under additional conditions.
Existence of the Mean
A finite expected value is fundamental to standard versions of the theorem. Some extremely heavy-tailed distributions, such as the Cauchy distribution, do not have a finite mean. Their sample averages do not converge to a population mean in the usual way.
Random Error Versus Systematic Bias
A larger sample reduces random sampling variation. It does not automatically remove:
- Selection bias
- Nonresponse bias
- Measurement error
- Confounding
- Data-processing errors
- A poorly defined target population
A very large biased sample can produce an extremely precise estimate of the wrong quantity.
Law of Large Numbers Versus Central Limit Theorem
These two results answer different questions.
| Result | Main question | Conclusion |
|---|---|---|
| Law of large numbers | Does the statistic approach its expected value? | \(\bar{X}_n\to\mu\) |
| Central limit theorem | What is the approximate shape of its sampling distribution? | Standardized \(\bar{X}_n\) approaches \(N(0,1)\) |
The law of large numbers establishes consistency. The central limit theorem describes the remaining fluctuations around the target.
The central limit theorem gives\[ \frac{\bar{X}_n-\mu}{\sigma/\sqrt{n}} \xrightarrow{d} N(0,1), \]
while the law of large numbers gives\[ \bar{X}_n-\mu\longrightarrow0. \]
Both results reflect the importance of increasing sample size, but they describe different aspects of sampling behavior.
Key Takeaway
The law of large numbers states that sample averages and sample proportions approach their expected population values as the sample size increases:\[ \bar{X}_n\longrightarrow\mu \]
and\[ \hat{p}\longrightarrow p. \]
The standard error\[ SE(\bar{X}_n)=\frac{\sigma}{\sqrt{n}} \]
shows why larger samples produce more stable averages.
For sums, absolute variability increases because\[ SE(S_n)=\sigma\sqrt{n}, \]
but the normalized sum still satisfies\[ \frac{S_n}{n}\longrightarrow\mu. \]
The law also explains why empirical relative frequencies—and more generally empirical distributions—approach their underlying theoretical probabilities. It reduces random sampling error, but it cannot correct systematic bias or poor data collection.