Sample Mean

The sample mean is the average of the observations you have, \(\bar{x} = \frac{1}{n}\sum_{i=1}^{n} x_i\). It is a statistic: it changes from sample to sample, and its job is to estimate the true mean of the population, which does not change and is not observed.

Five test scores of 80, 85, 90, 95 and 100 have a sample mean of 90. Ten people with a combined height of 1,720 centimeters have a sample mean of 172. Neither number is the population value; each is one draw from a distribution of possible answers.

What has to be true for it to work

The sample mean is unbiased: \(E[\bar{x}] = \mu\), so across repeated samples it is centered on the quantity it estimates. That property is not automatic — it holds when the sample is drawn at random from the population in question, and fails when the way units enter the sample is related to their values. A survey answered mostly by enthusiasts, a log that drops sessions which crashed, or a cohort defined by having survived long enough to be measured all produce means that are centered on something other than \(\mu\), and no sample size repairs it.

For independent draws from a distribution with finite variance, the spread of the sample mean across samples is the standard error, \(\sigma/\sqrt{n}\). Both conditions do the work. Under dependence the variance of the mean picks up the covariance terms, \(\operatorname{Var}(\bar{x}) = \left(\sum \operatorname{Var} + 2\sum \operatorname{Cov}\right)/n^2\), so the independent formula is wrong in whichever direction the covariances run: positive correlation — clustering, autocorrelated time series — makes it understate the spread, often badly, while negative correlation makes it overstate it. Either way the design’s own variance has to be used.

The \(\sqrt{n}\) is the whole economics of measurement: four times the data halves the uncertainty. In practice \(\sigma\) is unknown and is replaced by the sample standard deviation, giving \(s/\sqrt{n}\); that substitution adds variability of its own, which is why the reference distribution is Student’s t rather than the normal. The \(t\) result is exact only for independent draws from a normal distribution; outside that it is a large-sample approximation whose quality depends on the shape of the distribution and on \(n\). The substitution alone does not deliver it — with two Bernoulli(0.5) draws, \(s\) is zero half the time and the statistic has no \(t\) distribution at all. See t-test and standard error.

Exactly normal, or approximately

If the population is normal, the sample mean is exactly normal with variance \(\sigma^2/n\) at every sample size. If it is not, the central limit theorem says the sample mean approaches normality as \(n\) grows, and the usual shorthand is that thirty observations are enough.

That shorthand is badly wrong for skewed data, which is most business data. The calculation below builds a 95 percent t interval around the sample mean and counts how often it actually contains the true mean.

import numpy as np
from scipy import stats

def interval_coverage(draw, true_mean, n, replicates=100_000, seed=8):
    """How often the usual t interval around the sample mean contains the true mean."""
    rng = np.random.default_rng(seed)
    samples = draw(rng, (replicates, n))
    means = samples.mean(axis=1)
    errors = samples.std(axis=1, ddof=1) / np.sqrt(n)
    half = stats.t.ppf(0.975, n - 1) * errors
    return float(((means - half <= true_mean) & (true_mean <= means + half)).mean())

normal = lambda rng, size: rng.normal(100, 15, size)
# A right-skewed variable, the shape revenue and session length usually take.
lognormal = lambda rng, size: rng.lognormal(4.0, 1.2, size)
skewed_mean = np.exp(4.0 + 1.2**2 / 2)

for n in (10, 30, 100, 1000):
    print(f"{n:>4} {interval_coverage(normal, 100.0, n):.4f} "
          f"{interval_coverage(lognormal, skewed_mean, n):.4f}")
# 10 0.9507 0.7952
# 30 0.9497 0.8476
# 100 0.9507 0.8956
# 1000 0.9501 0.9379

sample = np.array([80, 85, 90, 95, 100])
# One large value moves the mean and barely moves the median.
with_outlier = np.append(sample, 1000)
print(f"{sample.mean():.1f} {np.median(sample):.1f} "
      f"{with_outlier.mean():.1f} {np.median(with_outlier):.1f}")
# 90.0 90.0 241.7 92.5

On normal data the interval is exact at every size. On a right-skewed variable of the kind revenue, session length and time-to-event produce, a “95 percent” interval covers 85 percent of the time at \(n = 30\) and has still not arrived at 95 by a thousand observations. The intervals are also mis-centered rather than merely too short: with a long right tail, most samples miss the mean downward and a few overshoot badly.

What to do about it is a design question rather than a formula question. Use a transformation, a bootstrap interval, or a quantity that is better behaved than the mean — and if the mean is the quantity the decision needs, plan for far more data than the rule of thumb implies. Incremental revenue shows the same effect in an experiment, where a clearly detectable conversion effect sits next to a money effect that cannot be resolved at all.

One value can move it

The second half of the calculation adds a single large value to those five test scores. The mean goes from 90 to 241.7; the median goes from 90 to 92.5. The mean uses every observation, which is what makes it efficient on well-behaved data and fragile on data with a tail or a data-entry error.

The alternatives are robust and they are not free. A median or a trimmed mean is far less disturbed by extreme values, and it estimates a different quantity — the middle of the distribution, or the mean of a trimmed version of it. When the decision depends on totals, as revenue decisions do, that different quantity is not a substitute: total revenue is the mean times the number of customers, and the median has no such relationship. Choosing between them is covered under mean.

References

• NIST/SEMATECH e-Handbook of Statistical Methods, section 7.2.2 Are the data consistent with the assumed process mean?
• Lawrence D. Brown, T. Tony Cai and Anirban DasGupta, “Interval Estimation for a Binomial Proportion”, Statistical Science 16(2), 101–133, 2001