Margin of Error (MoE)

For an interval reported symmetrically around an estimate, a margin of error is its half-width: the “plus or minus” amount. Asymmetric intervals should report their endpoints or separate lower and upper errors; one centered ± value need not reproduce them. The definition that matters is the one the US Census Bureau uses, because of the clause at the end — a margin of error “describes the precision of an ACS estimate at a given level of confidence.“

Without a confidence level, the quantity is undefined. A ±3 with no level attached could be almost any multiple of the underlying variability, which is why the first question to ask of any such figure is what level it was computed at.

It is a multiple of the standard error

The two quantities are related by exactly one number. A standard error “measures the variability of an estimate due to sampling and provides the basis for calculating the MOE,” and the Census Bureau states its own conversion: its published margins are at 90% confidence, so a user recovers the standard error by “dividing the positive value of its MOE by 1.645.”

Two consequences. Quoting a standard error as though it were a margin of error understates the uncertainty by the multiplier — a factor of about two at 95%. And two margins at different levels are not comparable: a 90% figure looks tighter than a 95% figure on identical data, which is worth checking before concluding that one survey is more precise than another. The level is chosen, and it is the inverse of a significance level.

The Census Bureau’s worked example makes the whole chain concrete: 564,757 one-person households in Colorado in 2015 with a margin of error of 10,127, giving a 90% interval of 554,630 to 574,884 — and a standard error of about 6,156. The interval is “a range expected to contain the true or population value of an estimate 90 percent of the time.” See confidence intervals for what that phrasing does and does not license.

Reading a reported margin

For a proportion the margin at 95% is about \(1.96\sqrt{\hat{p}(1-\hat{p})/n}\), which is largest near \(\hat{p} = 0.5\) and shrinks like \(1/\sqrt{n}\). Three things follow that are routinely missed in reporting.

• Halving the margin costs four times the sample. The quadratic relationship is why precision gets expensive quickly — the same arithmetic as in power analysis.
• Subgroups have much larger margins than the headline. A margin quoted for a whole sample says nothing about a slice of it, and a slice a tenth the size has a margin roughly three times as wide.
• Overlapping margins do not settle a comparison. “52% versus 48%, margin of error ±3” is not a tie and not a lead; the quantity that decides is an interval for the difference, which has its own standard error. Comparing two intervals by eye is the most common error in poll coverage.

Where estimates are combined — a ratio, a sum, a year-over-year change — margins do not simply add. The Census Bureau’s own remark about its replicate method names why: “unlike available approximation formulas, this method results in an exact MOE by incorporating the covariance.” Approximations that ignore the covariance between the parts can be badly wrong in either direction.

What it does not include

Sampling variability only. The same handbook separates out nonsampling error — problems “in the sampling frame or survey questionnaires, mistakes in how the data are reported or coded, issues related to data processing or weighting, or problems related to interviewer bias or nonresponse bias” — and states that “the error measures and statistical tests described in this section do not tell us about the magnitude of nonsampling errors.”

So a ±1% margin on a badly worded question is a precise measurement of the wrong thing. In product analytics the equivalent is a tight margin computed over a row count rather than over independent users: ten thousand events from two thousand people is not \(n = 10{,}000\) for a question about people, and the margin computed that way is too small. Check the unit of sampling before quoting the number — see sample size and IID.

References

• NIST/SEMATECH e-Handbook of Statistical Methods, §7.1.4 “What are confidence intervals?”
• U.S. Census Bureau, “Understanding Error and Determining Statistical Significance” (2020).