The Central Limit Theorem and Normal Approximation

The central limit theorem explains why the normal distribution appears so frequently in statistics. Even when individual observations have a strongly non-normal distribution, averages and sums of many independent observations often have approximately normal sampling distributions.

From a Binomial Distribution to a Normal Curve

Consider an online game in which the probability of winning a small prize is\[ p=0.2. \]

Suppose the game is played independently \(n\) times. Define\[ X=\text{number of small prizes won}. \]

Each play is a Bernoulli trial:\[ Y_i= \begin{cases} 1, & \text{if a small prize is won},\\ 0, & \text{otherwise}. \end{cases} \]

The total number of small prizes is\[ X=Y_1+Y_2+\cdots+Y_n. \]

Because the trials are independent and have the same success probability,\[ X\sim\operatorname{Binomial}(n,0.2). \]

The shape of this distribution changes as \(n\) increases.

• When \(n=1\), the distribution contains only 0 and 1 and is strongly asymmetric.
• When \(n=10\), the distribution is still right-skewed.
• When \(n=50\), it begins to resemble a bell-shaped normal curve.
• As \(n\) becomes larger, the normal approximation generally improves.

This behavior illustrates the central limit theorem.

What the Central Limit Theorem Says

Suppose \(Y_1,Y_2,\ldots,Y_n\) are independent and identically distributed random variables with:\[ E(Y_i)=\mu \]

and\[ SD(Y_i)=\sigma, \]

where \(\sigma\) is finite.

Define the sample mean as\[ \bar{Y} = \frac{Y_1+Y_2+\cdots+Y_n}{n}. \]

The central limit theorem states that, as \(n\) becomes large, the standardized sample mean\[ Z = \frac{\bar{Y}-\mu}{\sigma/\sqrt{n}} \]

converges in distribution to a standard normal random variable:\[ Z \xrightarrow{d} N(0,1). \]

In practical terms,\[ \bar{Y} \approx N\left(\mu,\frac{\sigma^2}{n}\right) \]

for sufficiently large \(n\).

Equivalently, the sample mean has approximately:\[ E(\bar{Y})=\mu \]

and\[ SE(\bar{Y})=\frac{\sigma}{\sqrt{n}}. \]

The central limit theorem concerns the sampling distribution of a statistic—not necessarily the distribution of the individual observations.

The Sample Sum

The theorem also applies to the sum\[ S_n=Y_1+Y_2+\cdots+Y_n. \]

Because\[ S_n=n\bar{Y}, \]

its expected value is\[ E(S_n)=n\mu, \]

and its standard error is\[ SE(S_n)=\sigma\sqrt{n}. \]

For sufficiently large \(n\),\[ S_n \approx N(n\mu,n\sigma^2). \]

The standardized sum is therefore\[ Z = \frac{S_n-n\mu}{\sigma\sqrt{n}}. \]

This standardized statistic is approximately distributed as \(N(0,1)\).

Sample Percentages

A sample proportion is the average of 0–1 indicator variables. If\[ P(Y_i=1)=p, \]

then\[ E(Y_i)=p \]

and\[ SD(Y_i)=\sqrt{p(1-p)}. \]

The sample proportion is\[ \hat{p} = \frac{Y_1+\cdots+Y_n}{n}. \]

Its expected value and standard error are\[ E(\hat{p})=p \]

and\[ SE(\hat{p}) = \sqrt{\frac{p(1-p)}{n}}. \]

For sufficiently large \(n\),\[ \hat{p} \approx N\left( p,\frac{p(1-p)}{n} \right). \]

A percentage is simply \(100\hat{p}\), so\[ E(100\hat{p})=100p \]

and\[ SE(100\hat{p}) = 100\sqrt{\frac{p(1-p)}{n}} \]

percentage points.

Standardization

Normal approximation begins by standardizing the statistic:\[ Z = \frac{\text{observed statistic}-\text{expected value}} {\text{standard error}}. \]

For a sample mean,\[ Z = \frac{\bar{Y}-\mu}{\sigma/\sqrt{n}}. \]

For a sample sum,\[ Z = \frac{S_n-n\mu}{\sigma\sqrt{n}}. \]

For a sample proportion,\[ Z = \frac{\hat{p}-p} {\sqrt{p(1-p)/n}}. \]

The resulting \(z\)-score measures how many standard errors the statistic is above or below its expected value.

Once the statistic has been standardized, probabilities can be approximated using the standard normal distribution.

Why the Theorem Is Remarkable

The distribution of the original observations does not have to be normal. It may be:

• Skewed
• Discrete
• Multimodal
• Asymmetric
• Otherwise irregular

Under appropriate conditions, the sampling distribution of the mean or sum still approaches a normal distribution as the sample size increases.

This does not mean the original population becomes normal. It means the distribution of a statistic calculated from repeated samples becomes approximately normal.

Example: A Right-Skewed Income Distribution

Suppose annual household income has a strongly right-skewed population distribution with:\[ \mu=\$67{,}000 \]

and\[ \sigma=\$38{,}000. \]

The distribution of individual household incomes is far from normal. However, consider taking independent random samples of \(n=100\) households and calculating the average income in each sample.

The expected value of the sample mean is\[ E(\bar{X})=\$67{,}000. \]

Its standard error is\[ SE(\bar{X}) = \frac{\$38{,}000}{\sqrt{100}} = \frac{\$38{,}000}{10} = \$3{,}800. \]

If the central limit theorem provides an adequate approximation, then\[ \bar{X} \approx N\left(\$67{,}000,\$3{,}800^2\right). \]

One standard error above the mean is\[ \$67{,}000+\$3{,}800 = \$70{,}800. \]

We want to approximate\[ P(\bar{X}>\$70{,}800). \]

Standardizing gives\[ z = \frac{70{,}800-67{,}000}{3{,}800} = 1. \]

Therefore,\[ P(\bar{X}>\$70{,}800) \approx P(Z>1) \approx 0.1587. \]

Thus, approximately 15.9% of samples of 100 households would have a sample mean above $70,800.

An Important Qualification

The central limit theorem is an asymptotic result: it describes what happens as \(n\) grows. It does not guarantee that every sample size labeled “large” gives an equally accurate approximation.

Because income distributions can be extremely right-skewed and may contain very large outliers, a sample size of 100 might provide only a rough approximation in some populations. The necessary sample size depends on the shape and tail behavior of the underlying distribution.

Binomial Counts as Sums

Return to the online game with success probability\[ p=0.2. \]

For one play, the indicator variable \(Y_i\) has expected value\[ E(Y_i)=p=0.2 \]

and variance\[ \operatorname{Var}(Y_i)=p(1-p)=0.2(0.8)=0.16. \]

Its standard deviation is\[ SD(Y_i)=\sqrt{0.16}=0.4. \]

For \(n\) independent plays, the number of small prizes is\[ X=Y_1+\cdots+Y_n. \]

Therefore,\[ E(X)=np \]

and\[ SE(X)=\sqrt{np(1-p)}. \]

For \(p=0.2\),\[ E(X)=0.2n \]

and\[ SE(X)=\sqrt{0.16n}=0.4\sqrt{n}. \]

These are exactly the mean and standard deviation formulas for a binomial random variable.

Example with 50 Plays

Suppose the game is played 50 times. Then\[ X\sim\operatorname{Binomial}(50,0.2). \]

The expected number of small prizes is\[ E(X)=50(0.2)=10. \]

The standard error is\[ SE(X) = \sqrt{50(0.2)(0.8)} = \sqrt{8} \approx 2.83. \]

The normal approximation is therefore\[ X\approx N(10,8). \]

Equivalently, the approximating normal distribution has mean 10 and standard deviation approximately 2.83.

Continuity Correction

A binomial random variable is discrete, while a normal random variable is continuous. A continuity correction improves the approximation by shifting integer boundaries by \(0.5\).

For example, suppose we want\[ P(X\leq 12). \]

Using a continuity correction, this becomes\[ P(X\leq12) \approx P(Y<12.5), \]

where\[ Y\sim N(10,8). \]

Standardizing,\[ z = \frac{12.5-10}{2.83} \approx 0.88. \]

Therefore,\[ P(X\leq12) \approx \Phi(0.88) \approx 0.81. \]

Without the continuity correction, we would use 12 instead of 12.5:\[ z = \frac{12-10}{2.83} \approx 0.71, \]

which gives a less accurate approximation of approximately 0.76.

When Is the Normal Approximation to a Binomial Reasonable?

A commonly used guideline is that both expected counts should be sufficiently large:\[ np\geq10 \]

and\[ n(1-p)\geq10. \]

Some introductory treatments use 5 instead of 10, but 10 is a more conservative guideline.

For \(n=50\) and \(p=0.2\),\[ np=10 \]

and\[ n(1-p)=40. \]

The guideline is satisfied, although the distribution may retain some right skew because \(p\) is relatively far from \(0.5\).

When \(p\) is very close to 0 or 1, a much larger sample may be needed.

The Central Limit Theorem Does Not Say Everything Is Normal

Several common misunderstandings should be avoided.

The Population Does Not Become Normal

A skewed population remains skewed. It is the sampling distribution of the mean or sum that becomes approximately normal.

One Sample Does Not Have a Sampling Distribution

A single sample produces one observed statistic. The sampling distribution describes how that statistic would vary across repeated samples.

Large \(n\) Is Context-Dependent

There is no universal sample size at which the approximation suddenly becomes valid. Strong skewness, outliers, and heavy tails may require larger samples.

Finite Variance Matters

The standard form of the theorem assumes a finite population variance. Some heavy-tailed distributions do not satisfy this condition, and the conventional normal approximation may fail.

Independence Matters

Strong dependence among observations can invalidate the ordinary standard-error formulas and the basic independent-sample version of the theorem. Specialized central limit theorems exist for some dependent data, but they require additional assumptions.

Sampling Without Replacement

If observations are sampled without replacement from a finite population, they are not independent. When the sample is small relative to the population, the independent-sampling formulas remain good approximations.

When the sampling fraction is not negligible, the standard error of the sample mean should include the finite population correction:\[ SE(\bar{X}) = \frac{\sigma}{\sqrt{n}} \sqrt{\frac{N-n}{N-1}}, \]

where \(N\) is the population size.

The sampling distribution may still be approximately normal under suitable conditions, but its variability is smaller than it would be under independent sampling.

Summary of Central Limit Theorem Formulas

Statistic	Expected value	Standard error	Approximate distribution
Sample mean \(\bar{X}\)	\(\mu\)	\(\sigma/\sqrt{n}\)	\(N(\mu,\sigma^2/n)\)
Sample sum \(S_n\)	\(n\mu\)	\(\sigma\sqrt{n}\)	\(N(n\mu,n\sigma^2)\)
Binomial count \(X\)	\(np\)	\(\sqrt{np(1-p)}\)	\(N(np,np(1-p))\)
Sample proportion \(\hat{p}\)	\(p\)	\(\sqrt{p(1-p)/n}\)	\(N(p,p(1-p)/n)\)
Sample percentage	\(100p\%\)	\(100\sqrt{p(1-p)/n}\)	Approximately normal

Key Takeaway

The central limit theorem states that sums and averages of many independent, identically distributed observations with finite variance have approximately normal sampling distributions as the sample size grows.

For a sample mean,\[ Z = \frac{\bar{X}-\mu}{\sigma/\sqrt{n}} \]

is approximately standard normal. For a binomial count,\[ Z = \frac{X-np}{\sqrt{np(1-p)}} \]

is approximately standard normal when the expected numbers of successes and failures are sufficiently large.

The theorem is powerful because the original population does not need to be normal. However, the accuracy of the approximation depends on the sample size, the shape and tails of the underlying distribution, and whether the required independence and finite-variance conditions are reasonable.