Confidence Level

The confidence level is the coverage rate attached to an interval — 90%, 95%, 99% — and the NIST handbook is careful to say whose property it is: levels are “selected by the user,” and the meaning is about repetition:

If the same population is sampled on numerous occasions and interval estimates are made on each occasion, the resulting intervals would bracket the true population parameter in approximately 95 % of the cases.

So the level describes the procedure, not the interval you are holding. An observed interval either contains the true value or does not. Increasing the level targets higher coverage across repeated samples; for the nested z intervals below, it produces a wider interval from the same data. That frequentist coverage statement is not a posterior probability for the particular interval.

It is the same dial as α

The handbook states the correspondence directly: “a confidence stated at a \(1-\alpha\) level can be thought of as the inverse of a significance level, \(\alpha\).” Choosing 95% confidence is choosing \(\alpha = 0.05\), and it fixes the multiplier in the interval

\[\bar{Y} \pm z_{1-\alpha/2}\,\frac{\sigma}{\sqrt{N}}\]

which is where the familiar constants come from — 1.645 at 90%, 1.96 at 95%, 2.576 at 99%. So a level is not a statement about how good your data is; it is the same error-tolerance decision made at significance level, expressed from the other side. See confidence intervals and margin of error, whose value changes entirely with the level chosen.

Two reporting rules that follow

1. Always state the level. A “±3” with no level is undefined — the same data supports ±2.5 at 90% and ±3.9 at 99%. A figure without its level cannot be interpreted or compared.
2. Never compare intervals at different levels. A 90% interval looks tighter than a 95% interval computed from identical data, so “our survey is more precise” may only mean “we chose a lower level.” Worth checking before believing a comparison.

Two further properties of the level are easy to miss. Intervals can be one-sided — “a one-sided confidence interval brackets the population parameter either from above or below” — which uses the whole \(\alpha\) in one tail and is the right form when only one direction matters. And when the standard deviation is estimated rather than known, intervals at the same level are “by necessity wider… because of the uncertainty in this estimate,” which is where \(t\) replaces \(z\).

Choosing one

There is nothing in the mathematics that prefers 95%. The level encodes how much you mind being wrong in the direction the interval protects against, so a high-stakes irreversible decision argues for a higher level and an exploratory estimate for a lower one. Stating the reason once in a methods note is more useful than inheriting a default and defending it later.

The level describes sampling uncertainty under the method; it does not account for every measurement or selection problem. The U.S. Census Bureau’s 2018 ACS handbook, chapter 7, page 45, advises reporting interval bounds within the natural range of the quantity, such as zero for a population count whose calculated lower bound is negative. It also notes that negative values can be valid for income. That reporting guidance does not cure bias or poor coverage in the underlying method.

References

• NIST/SEMATECH e-Handbook of Statistical Methods, §7.1.4 “What are confidence intervals?”.

• U.S. Census Bureau, Understanding and Using American Community Survey Data: What All Data Users Need to Know (2018), chapter 7, p. 45.