Sampling Distributions of Statistics
When a statistic is calculated from a random sample, its value can change from one sample to another. A sampling distribution describes all the values the statistic could take and the probability associated with each value.
A Coin-Tossing Example
Suppose a fair coin is tossed 100 times, and let \(X\) be the number of tails.
The possible values of \(X\) are\[ 0,1,2,\ldots,100. \]
If we define a tail as a success, then each toss is a Bernoulli trial with\[ p=P(\text{tail})=0.5. \]
Because the 100 tosses are independent and have the same probability of success,\[ X\sim \operatorname{Binomial}(100,0.5). \]
Its probability mass function is\[ P(X=k) = \binom{100}{k}(0.5)^k(0.5)^{100-k}, \qquad k=0,1,\ldots,100. \]
Since both factors involving \(0.5\) can be combined,\[ P(X=k) = \binom{100}{k}(0.5)^{100}. \]
This formula gives the probability of observing exactly \(k\) tails.
From Data to a Statistic
The complete outcome of the experiment is a sequence of 100 coin tosses. One possible result might begin as\[ (\text{tail},\text{head},\text{tail},\text{tail},\ldots). \]
The statistic \(X\) reduces that full sequence to a single number: the total number of tails.
Different sequences can produce the same statistic. For example, each of the following contains two tails:
- Tail, tail, head
- Tail, head, tail
- Head, tail, tail
The statistic ignores the order and records only the count.
A statistic is a numerical function of the observed sample.
Because the sample is random, the statistic calculated from it is also random before the data are observed. It is therefore a random variable.
What Is a Sampling Distribution?
The sampling distribution of a statistic is the probability distribution of that statistic over repeated samples generated by the same process.
For the coin-tossing example:
- The sampling process is tossing a fair coin 100 times.
- The statistic is the number of tails.
- The sampling distribution is \(\operatorname{Binomial}(100,0.5)\).
Conceptually, imagine repeating the entire 100-toss experiment many times:
- Toss the coin 100 times.
- Count the number of tails.
- Record that count.
- Repeat the process.
The resulting counts might look like\[ (47,53,50,44,56,49,\ldots). \]
A histogram of many such counts would approach the theoretical binomial probability histogram.
Probability Histogram of the Statistic
A probability histogram displays the sampling distribution:
- The horizontal axis shows possible values of the statistic.
- The height of each bar shows the probability of that value.
- The total area of all bars equals 1.
For \(X\sim\operatorname{Binomial}(100,0.5)\), the histogram is symmetric and centered at 50.
Values near 50 are relatively likely because there are many ways to arrange roughly equal numbers of heads and tails. Extreme results, such as 0 tails or 100 tails, are possible but extremely unlikely.
For example,\[ P(X=50) = \binom{100}{50}(0.5)^{100} \approx 0.0796. \]
Thus, the probability of getting exactly 50 tails is approximately 7.96%.
In contrast,\[ P(X=0)=(0.5)^{100} \approx 7.89\times 10^{-31}. \]
An outcome of zero tails is possible, but its probability is extraordinarily small.
Expected Value of the Number of Tails
For a binomial random variable,\[ E(X)=np. \]
Therefore,\[ E(X)=100(0.5)=50. \]
This means that over many repetitions of the 100-toss experiment, the average number of tails will approach 50.
It does not mean that every experiment will produce exactly 50 tails.
Standard Error of the Number of Tails
The standard error of a statistic is the standard deviation of its sampling distribution.
For a binomial count,\[ SE(X)=\sqrt{np(1-p)}. \]
In this example,\[ SE(X) = \sqrt{100(0.5)(0.5)} = \sqrt{25} = 5. \]
Therefore, the number of tails typically differs from its expected value of 50 by approximately five tosses.
The standard error measures sample-to-sample variability in a statistic.
Because the sampling distribution is approximately bell-shaped, the empirical rule provides useful approximations:\[ P(45\lesssim X\lesssim55)\approx 68\% \]
and\[ P(40\lesssim X\lesssim60)\approx 95\%. \]
Because \(X\) is discrete, more accurate normal approximations use a continuity correction.
Sampling Distribution of the Proportion of Tails
Instead of counting tails, suppose we calculate the sample proportion\[ \hat{p}=\frac{X}{100}. \]
The possible values are\[ 0,0.01,0.02,\ldots,1. \]
Its expected value is\[ E(\hat{p})=p=0.5. \]
Its standard error is\[ SE(\hat{p}) = \sqrt{\frac{p(1-p)}{n}}. \]
Therefore,\[ SE(\hat{p}) = \sqrt{\frac{0.5(0.5)}{100}} = 0.05. \]
The sampling distributions of \(X\) and \(\hat{p}\) contain the same fundamental information because\[ \hat{p}=\frac{X}{100}. \]
They simply use different scales:
| Statistic | Expected value | Standard error |
|---|---|---|
| Number of tails \(X\) | 50 | 5 tails |
| Proportion of tails \(\hat{p}\) | 0.50 | 0.05 |
| Percentage of tails \(100\hat{p}\) | 50% | 5 percentage points |
Why Expected Value and Standard Error Are Not Enough
The expected value and standard error provide two important summaries:
- The expected value identifies the center.
- The standard error describes the typical amount of variation.
However, they do not completely determine a distribution in general. Two sampling distributions can have the same expected value and standard error while having different:
- Shapes
- Tail probabilities
- Degrees of symmetry
- Probabilities of extreme outcomes
- Numbers of peaks
The complete sampling distribution provides more detailed information. It allows us to calculate probabilities such as\[ P(X=50), \]\[ P(X\leq40), \]
or\[ P(45\leq X\leq55). \]
For example, the probability of obtaining between 45 and 55 tails, inclusive, is\[ P(45\leq X\leq55) = \sum_{k=45}^{55} \binom{100}{k}(0.5)^{100}. \]
The expected value and standard error alone do not provide this exact probability. The full binomial sampling distribution does.
Statistic Versus Parameter
A parameter is a fixed numerical characteristic of a population or probability model. A statistic is calculated from a sample and therefore varies across samples.
In this example:
- The probability of tails, \(p=0.5\), is a parameter.
- The observed number of tails, \(X\), is a statistic.
- The observed proportion of tails, \(\hat{p}\), is another statistic.
Before tossing the coins, \(X\) and \(\hat{p}\) are random variables. After observing the results, they become known numerical values.
Suppose the experiment produces 47 tails. We then observe\[ X=47 \]
and\[ \hat{p}=0.47. \]
The sampling distribution describes how these statistics could have behaved before the particular sample was observed.
Probability Distribution Versus Empirical Distribution
A theoretical sampling distribution should be distinguished from a histogram created from observed simulations.
The binomial formula defines the theoretical distribution:\[ P(X=k) = \binom{100}{k}(0.5)^{100}. \]
A simulation approximates this distribution by repeatedly tossing coins or generating computer-based random outcomes.
If the experiment is simulated 10,000 times, the empirical proportion of simulations with exactly 50 tails should be close to\[ P(X=50)\approx 0.0796. \]
The empirical histogram will not match the theoretical histogram perfectly because simulation itself is subject to random variation. The match generally improves as the number of simulations increases.
Sampling Distributions in Statistical Inference
Sampling distributions are central to statistical inference because they tell us how much a statistic would vary under repeated sampling.
They are used to construct:
- Margins of error
- Confidence intervals
- Hypothesis tests
- P-values
- Standard errors
- Statistical significance measures
For example, if a fair coin produces 70 tails in 100 tosses, the sampling distribution helps determine whether that result is reasonably compatible with \(p=0.5\) or is unusually far from what the fair-coin model predicts.
The standardized value would be\[ z = \frac{70-50}{5} = 4. \]
The observed count is four standard errors above its expected value, making it extremely unusual under the fair-coin model.
Key Takeaway
A sampling distribution describes the possible values of a statistic and their probabilities across repeated samples.
For the number of tails in 100 tosses of a fair coin,\[ X\sim\operatorname{Binomial}(100,0.5), \]
with\[ E(X)=50 \]
and\[ SE(X)=5. \]
The expected value and standard error summarize the center and spread, but the complete sampling distribution provides more information about shape, exact probabilities, and unusual outcomes. This makes sampling distributions a foundation for confidence intervals, hypothesis testing, and statistical inference.