Type I Error

A Type I error is rejecting a null hypothesis that is in fact true — a false positive. A valid test at significance level \(\alpha\) controls this probability at or below that level under its assumptions. Discrete or conservative tests can have a lower actual rate: in the NIST handbook’s words, a test is built so that “the risk of rejecting the null hypothesis, when it is in fact true, is small. This risk, \(\alpha\), is often referred to as the significance level of the test.”

The term only makes sense next to its counterpart, so here is the whole grid.

One rate is chosen, the other is not

This asymmetry is the practical heart of the term. \(\alpha\) is set by you in advance. \(\beta\), the handbook says, “is not chosen by the user but is determined, as one might expect, by the magnitude of the real discrepancy” — “large discrepancies between reality and the null hypothesis are easier to detect and lead to small errors of the second kind; while small discrepancies are more difficult to detect and lead to large errors of the second kind.”

And the two move against each other: “the risk \(\beta\) increases as the risk \(\alpha\) decreases.” Which makes the common instinct — tighten \(\alpha\) to be safe — a decision rather than a precaution. Tightening it trades a lower false-positive rate for a higher chance of missing something real, at fixed sample size.

So “being conservative” needs a direction. Conservative about shipping a change that might not work means a small \(\alpha\). Conservative about abandoning a change that actually works means a small \(\beta\). Both are defensible and they pull opposite ways, so the useful question is which mistake is more expensive here — and that is a domain judgment, not a statistical one.

At fixed \(\alpha\), a larger independent sample can lower \(\beta\), while the test continues to control Type I error at the chosen level under its assumptions. More information, including variance reduction, may let you lower \(\alpha\) while retaining target power, but that requires an explicit redesign. See power analysis and sample size.

The rate is per test

\(\alpha\) bounds one test’s false-positive rate, and the bound does not extend to a collection of them. With \(k\) independent tests at level \(\alpha\), the probability of at least one false positive is \(1-(1-\alpha)^k\): about 0.23 at five tests, 0.40 at ten, 0.64 at twenty, for \(\alpha = 0.05\).

Everyday practices multiply \(k\) without anyone deciding to. Breaking a result down by segment, watching several guardrail metrics, checking an experiment daily, running several variants at once — each adds comparisons. The first step is not a correction formula; it is knowing the count. Then you choose what to control: the chance of any false positive across the family, or the share of false positives among the results you act on. See risk of peeking for the repeated-looks version and multivariate test for the many-arms version.

What it does not tell you

Two things worth separating from \(\alpha\). It is not the probability that a particular significant result is wrong — that depends on how often the null is actually true among the things you test, which \(\alpha\) says nothing about. And it says nothing about size: a correctly rejected null can correspond to a difference too small to act on, which is why effect size is reported alongside, and why trivial effects is a distinct problem from false positives.

Both are reasons to report an interval rather than a verdict. A confidence interval at level \(1-\alpha\) carries the same error rate and the magnitude together, and it makes a “significant but tiny” result visible as what it is — see hypothesis testing for how the pieces fit.

References

• NIST/SEMATECH e-Handbook of Statistical Methods, §7.1.3 “What are statistical tests?”
• §7.1.4 “What are confidence intervals?”
• American Statistical Association, statement on statistical significance and p-values