Standard Error (SE)
The standard error of an estimate is how much that estimate would move around if you drew the sample again. In the US Census Bureau’s words, it “measures the variability of an estimate due to sampling” and “provides a quantitative measure of the extent to which an estimate derived from a sample can be expected to deviate from the value for the full population.”
Note what the subject of that sentence is. A standard error is a property of the estimate, not of the data — which is the whole difference between it and a standard deviation, and the source of most of the confusion around it.
Standard deviation or standard error
| Describes | What more data does to it | |
|---|---|---|
| Standard deviation (\(\sigma\) or \(s\)) | How spread out the individual observations are | Nothing in particular — it estimates a fixed property of the population |
| Standard error | How spread out the estimate would be across repeated samples | Shrinks, because a bigger sample pins the estimate down |
For a sample mean and a proportion the usual formulas are
\[SE_{\bar{x}} = \frac{s}{\sqrt{n}} \qquad SE_{\hat{p}} = \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}\]
where \(s\) is the sample standard deviation, \(\hat{p}\) the sample proportion, and \(n\) the sample size. Both shrink like \(1/\sqrt{n}\), which is why quadrupling a sample halves the standard error rather than quartering it.
A standard error is not a margin of error
This is the most common error in reporting, and it is easy to see why: both are “plus or minus” quantities attached to an estimate. They are not interchangeable. A margin of error, again in the Census Bureau’s definition, “describes the precision of an ACS estimate at a given level of confidence.” A standard error has no confidence level attached; it “provides the basis for calculating the MOE.”
The relationship is a multiplier, and the Census Bureau states its own: its published margins are at 90% confidence, so a user recovers the standard error by “dividing the positive value of its MOE by 1.645.” The bureau’s worked example: 564,757 one-person households in Colorado in 2015 with a margin of error of 10,127, giving a 90% interval of 554,630 to 574,884 — and a standard error of about 6,156.
Run it the other way with a survey figure. Of 1,000 people, 520 say yes, so \(\hat{p} = 0.52\) and
\[SE = \sqrt{\frac{0.52 \times 0.48}{1000}} = 0.0158\]
which is 1.58 percentage points. That is the standard error. A 95% margin of error, using the normal multiplier, is \(1.96 \times 0.0158 \approx 0.031\) — about 3.10 percentage points, giving roughly 48.9% to 55.1%.
So calling the standard error a margin of error understates the uncertainty by exactly the multiplier — a factor of 1.96 at 95%, 1.645 at 90%. It is how a 52-to-48 result gets described as decisive when the interval covers both sides of 50%. If you see a “±” figure with no confidence level stated, you cannot tell which quantity it is, and the two differ by about a factor of two.
What dividing by \(\sqrt{n}\) assumes
Both formulas above divide by \(\sqrt{n}\), and that step assumes the observations are independent draws. Where they are not, the formula is not the right one — and it does not warn you, because it still returns a number.
Which direction it is wrong in is set by the sign of the covariance, not by dependence as such. Two observations with variance 1 each and covariance \(-0.5\) give a mean with variance \((1 + 1 - 1)/4 = 0.25\) and a standard error of 0.5, against about 0.707 if they were independent — so negative correlation makes the true standard error smaller, and some designs induce that deliberately. Positive correlation makes it larger.
Positive dependence is common in repeated measurements of people and in household or site clusters, and ignoring it can understate uncertainty. It is not universal: consecutive observations can have negative or mixed covariance. Use the actual design and covariance structure to estimate uncertainty rather than assuming the direction. The Census Bureau’s own remark about its variance replicate method makes the same point from the other side: “unlike available approximation formulas, this method results in an exact MOE by incorporating the covariance.”
Two ways this goes wrong in company data are worth naming.
- Using a row count as \(n\). Ten thousand events from two thousand users is not \(n = 10{,}000\) for a question about users. Repeated observations of the same person carry less information than ten thousand different people would, so the standard error computed from the row count is too small — often by a lot. See sample size.
- Treating time series rows as draws. Consecutive days are correlated, which is the ordinary case rather than the exception — see temporal autocorrelation.
So the question to ask before quoting a standard error is: what is the unit being sampled, and is \(n\) the number of independent such units? If the answer is no, the figure needs a method that matches the design — a clustered or weighted sample needs a standard error that accounts for the design — rather than a caveat in a footnote. Where a closed form is awkward, resampling is the general-purpose alternative: see bootstrap.
What it does not measure at all
A standard error quantifies sampling variability and nothing else. The Census Bureau separates it explicitly from what it calls nonsampling error, which “might also be introduced during any of the complex operations used to collect and process” the data — “problems in the sampling frame or survey questionnaires, mistakes in how the data are reported or coded, issues related to data processing or weighting, or problems related to interviewer bias or nonresponse bias.”
And it says what follows: nonsampling error “is very difficult to measure directly,” and “the error measures and statistical tests described in this section do not tell us about the magnitude of nonsampling errors.”
That is the sentence to carry over to your own data. A standard error computed from a tracking table says nothing about events the tracker dropped, users it double-counted, or a definition that changed in April. A tight interval around a wrong measurement is still wrong, which is why the quality of the measurement is a separate question from its precision — see data quality and metric definition.
Where it gets used
The standard error is the scale factor almost every other inferential quantity is built from, which is why getting it wrong propagates quietly.
- Intervals. Estimate plus or minus a multiplier times the standard error — see confidence intervals and margin of error, where the multiplier comes from.
- Test statistics. A difference divided by its standard error is what a z or t statistic is, so an understated standard error inflates the statistic and deflates the p-value.
- Experiment sizing. A target precision is a statement about a standard error, which is what makes it the lever in planning — see power analysis.
Further reading
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.