Sample Standard Deviation

The sample standard deviation, written \(s\), measures how far the values in a sample spread around their mean, in the units of the data. It is defined through the sample variance: the NIST handbook gives the variance as \(s^2 = \sum (Y_i - \bar{Y})^2/(N - 1)\), roughly the average squared distance from the mean, and the standard deviation as its square root. Taking the root matters for reading the number: “The standard deviation restores the units of the spread to the original data units (the variance squares the units).”

A three-value sample, 5, 7, 9, shows the arithmetic. The mean is 7. The deviations are -2, 0, and 2; squared, 4, 0, and 4; summed, 8. Dividing by \(n - 1 = 2\) gives a variance of 4, and the square root gives \(s = 2\). A value of zero would mean every observation was identical; the larger \(s\), the more the values differ from one another.

Why the denominator is n minus 1

The population standard deviation divides by the number of values. The sample version divides by one fewer, and the reason is that the deviations are measured from the sample mean rather than from the unknown population mean. The sample mean is the value that makes the squared deviations as small as they can be, so the deviations from it are slightly smaller, on average, than the deviations from the true mean would be; dividing by \(n\) would then underestimate the population variance. Dividing by \(n - 1\) corrects this exactly for the variance when the observations are independent draws from one population with finite variance: under that condition \(s^2\) is an unbiased estimator of \(\sigma^2\). The condition matters. The formula can always be computed as a description of the values in hand, but as an estimate of a population’s spread it depends on the sample being an IID sample from that population, and dependent observations, such as repeated readings from one unit, can give an \(s\) far from the population value in either direction. It does not make \(s\) an unbiased estimator of \(\sigma\), since the square root of an unbiased estimate is not itself unbiased, and \(s\) tends to sit slightly below \(\sigma\), noticeably so in very small samples. The correction is not a claim about the best estimator; it is a claim about the variance.

What it is, and what it is not

Because the deviations are squared before averaging, \(s\) is not the average distance from the mean, though it is often described that way. The handbook explains the effect of squaring: it gives greater weight to the values furthest from the mean, so that the variance, though meant as an overall measure of spread, can be dominated by the tails. The average distance is a different statistic, the average absolute deviation, which the same page describes as a measure that “does not square the distance from the mean, so it is less affected by extreme observations than are the variance and standard deviation.” The standard deviation is always at least as large as the average absolute deviation, and the gap between them grows with the weight in the tails. Nor is \(s\) the standard error: the standard deviation describes how the data spread, while the standard error of the mean, \(s/\sqrt{n}\), describes how precisely the mean has been estimated, and it shrinks with the sample where \(s\) does not. That formula also assumes independence; for clustered or time-ordered data the standard error has to account for the dependence, and \(s/\sqrt{n}\) is then the wrong number.

Its role in deciding how much data to collect

The place where the sample standard deviation most often changes a decision is before the data exist. Every sample size calculation for a mean or a difference in means needs an estimate of \(\sigma\), and the required sample scales with \(\sigma^2\). The arithmetic is unforgiving. In the usual normal-approximation formula, with the effect size, the significance level, and the target power held fixed, the required sample is proportional to \(\sigma^2\), so a standard deviation guessed at 70 percent of its true value gives a planned sample of about half what the study needs, since \(0.7^2 \approx 0.49\). The resulting power must be recalculated for the actual effect size, significance level, allocation, and test; it does not fall in direct proportion to the sample size. An undersized study is more likely to miss the effect it was designed to detect. The estimate has to come from somewhere, and the choices are a pilot sample, a previous study on a similar population, or a deliberately conservative upper bound; the power analysis entry covers how the number enters the calculation.

The estimate is itself uncertain, and more so than most people expect. For normally distributed data the confidence interval for \(\sigma\) follows from the chi-square distribution; NIST’s Dataplot reference gives the limits as \(s\sqrt{(n-1)/\chi^2_{1-\alpha/2,\,n-1}}\) and \(s\sqrt{(n-1)/\chi^2_{\alpha/2,\,n-1}}\) and warns that the interval “is highly sensitive to non-normality in the data”. With ten observations the 95 percent interval runs from about 0.69 to 1.83 times \(s\), which, squared, is a factor between roughly 0.5 and 3.3 on the required sample size; with thirty observations it is about 0.80 to 1.34 times \(s\), a factor between 0.6 and 1.8. When the plan rests on a small pilot, run the calculation as a sensitivity scenario at the upper end of that range rather than at the point estimate. If the standard deviation observed early in the real study is to be used to revise the sample size, decide in advance how and when that happens, since changing the size on the strength of interim results is itself a stopping rule.

• Is the number reported the standard deviation of the data or the standard error of an estimate?
• Which denominator was used, and does the software’s default match the intent?
• If the value feeds a sample-size calculation, where did it come from, and how many observations was it based on?
• Are there a few extreme values doing most of the work? If so, report the average absolute deviation or the interquartile range beside it.

References

• NIST/SEMATECH e-Handbook of Statistical Methods, section 1.3.5.6 Measures of Scale
• NIST, Dataplot Reference Manual: SD Confidence Limits