Significance Level (α)

The significance level \(\alpha\) is the false-positive rate you agree in advance to tolerate. In the NIST handbook’s phrasing, a test “is constructed so that the risk of rejecting the null hypothesis, when it is in fact true, is small. This risk, \(\alpha\), is often referred to as the significance level of the test.“

So \(\alpha\) is a decision about acceptable error, made before the data exist. It is not a property of your data, and it is not a measure of how much evidence you have.

It is a policy, not a discovery

Read the handbook’s reason for wanting \(\alpha\) small: “by having a test with a small value of \(\alpha\), we feel that we have actually ‘proved’ something when we reject the null hypothesis.” That is a statement about the standard of proof you are setting, not about the strength of any particular result.

Conventional levels follow this reading. 0.05 is a convention, 0.01 a stricter one for settings where a false positive is costly, 0.10 a looser one for exploratory work. None of them is a property of nature, and the ASA generalizes the point: conclusions “should not be based only on whether a p-value passes a specific threshold.” A result at \(p = 0.049\) and one at \(p = 0.051\) differ by nothing of substance.

Because \(\alpha\) is the rate at which a true null gets rejected, it is the same quantity as the Type I error rate. The threshold it implies on the test statistic is the critical value, and the comparison can be run either way — statistic against critical value, or p-value against \(\alpha\).

Lowering it is not free

Here is the fact that turns \(\alpha\) from a formality into a design decision. The other error — failing to reject a false null, with risk \(\beta\) — “is not chosen by the user but is determined… by the magnitude of the real discrepancy.” And the two are linked: “the risk \(\beta\) increases as the risk \(\alpha\) decreases.“

So moving \(\alpha\) from 0.05 to 0.01 does not make the test more careful in general. It makes one kind of mistake rarer and the other kind commoner, at fixed sample size. Being “conservative” about \(\alpha\) means accepting a higher chance of missing a real effect, which in a business context is often the more expensive error.

More information can ease the trade: with \(\alpha\) fixed, a larger independent sample or lower variance can reduce \(\beta\); lowering \(\alpha\) as well requires redesigning the test to retain the desired power — the calculation is in power analysis, the resulting detection probability is statistical power, and the effect a given design detects at a specified target power is its minimum detectable lift. Choosing \(\alpha\) therefore belongs with those decisions rather than being inherited as a default.

It is per test, and tests accumulate

\(\alpha\) bounds the false-positive rate of one test. Run many and the chance that at least one true null is rejected grows: with \(k\) independent tests at level \(\alpha\), that chance is \(1 - (1-\alpha)^k\), which at \(\alpha = 0.05\) reaches about 0.40 by ten tests and 0.64 by twenty.

This is why “we tested twenty variants and one was significant” is not a finding. Anywhere the count of tests grows — metric dashboards, per-segment breakdowns, repeated looks at a running experiment, several arms in one design — the per-test level stops describing the procedure. What to do about it is a choice of correction or of a different error criterion, but the first step is counting how many comparisons were actually made. See multivariate test for the many-arms case and risk of peeking for the repeated-looks case.

The interval on the other side of it

\(\alpha\) also sets the coverage of an interval. The handbook states the correspondence directly: “a confidence stated at a \(1-\alpha\) level can be thought of as the inverse of a significance level, \(\alpha\).” Choosing \(\alpha = 0.05\) is choosing 95% confidence intervals, and it fixes the multiplier in a margin of error.

Reporting the interval rather than the verdict is usually the better habit, because it carries the magnitude as well as the decision — and it makes the arbitrariness of the threshold visible instead of hiding it behind a yes or no. See confidence level for the same number read from the interval side.

Further reading

• NIST/SEMATECH e-Handbook of Statistical Methods, §7.1.3 “What are statistical tests?”
• §7.1.4 “What are confidence intervals?”
• American Statistical Association, statement on statistical significance and p-values