Hypothesis Testing

Hypothesis testing is a procedure for deciding whether data are inconsistent enough with a stated assumption to act against it. The NIST handbook puts the purpose in operational terms: a statistical test “provides a mechanism for making quantitative decisions about a process,” and “the intent is to determine whether there is enough evidence to ‘reject’ a conjecture or hypothesis about the process. The conjecture is called the null hypothesis.”

Note the verb. A test rejects or fails to reject; it does not confirm. That asymmetry is the source of nearly every misreading of a test result, so it is worth taking seriously from the start.

The five pieces

1. A null hypothesis \(H_0\) — the assumption being examined. “The mean linewidth is 500 micrometers.”
2. An alternative \(H_a\) — what you are guarding against. Two-sided if departures in either direction matter; one-sided if only one does.
3. A significance level \(\alpha\), fixed before looking.
4. A test statistic computed from the data.
5. A decision rule — the statistic is “tested against pre-determined upper and lower critical values,” and if it falls outside them “the null hypothesis is rejected.”

Equivalently the comparison is made with a p-value against \(\alpha\); the two formulations give the same verdict. Which specific test you use depends on the data and the question — see statistical tests for the families, and t-test, z-test, and two-proportion z-test for common cases.

Two errors, and only one is yours to choose

This is the structural fact that makes the procedure intelligible, and the handbook states it precisely.

	What it is	Who sets it
\(\alpha\) — Type I error	“The risk of rejecting the null hypothesis, when it is in fact true”	You do, in advance
\(\beta\) — error of the second kind	“The risk of failing to reject the null hypothesis when it is in fact false”	“Not chosen by the user but is determined… by the magnitude of the real discrepancy”

So the procedure gives you one dial and hands you the other risk as a consequence. And the two are coupled: “large discrepancies between reality and the null hypothesis are easier to detect and lead to small errors of the second kind; while small discrepancies are more difficult to detect and lead to large errors of the second kind. Also the risk \(\beta\) increases as the risk \(\alpha\) decreases.” Tightening \(\alpha\) to be careful makes you likelier to miss a real effect. What you can move without paying that price is the sample size — which is what power analysis computes and statistical power measures.

What “fail to reject” means

The handbook is unusually candid here, and the sentence deserves quoting in full because it names both readings: “not rejecting may be a good result if we want to continue to act as if we ‘believe’ the null hypothesis is true. Or it may be a disappointing result, possibly indicating we may not yet have enough data to ‘prove’ something by rejecting the null hypothesis.”

Two outcomes, one verdict. The p-value cannot tell them apart, and nothing in the test output distinguishes “no effect” from “not enough data.” What does distinguish them is asking what the study could have detected — the power question — and reporting an interval for the quantity instead of a verdict about a hypothesis. See confidence intervals.

You are testing a model, not a hypothesis

The ASA’s first principle names the object correctly: “p-values can indicate how incompatible the data are with a specified statistical model.” A statistical model is more than the absence of an effect — it carries the test’s assumptions about sampling, independence, and distributional shape.

So a small p-value is evidence against the package, and a violated assumption can produce one just as a real effect can. Clustered observations analyzed as independent, a stopping rule applied after peeking, a distribution with heavier tails than assumed — each can manufacture significance. Independence deserves explicit attention, especially when several rows share a user or another source; check the unit of sampling before selecting the test: see IID and standard error. Looking repeatedly at an accumulating result is its own version of the problem — see risk of peeking and stopping rules.

What the verdict does not include

Three things, each stated by the ASA. The result “does not measure the size of an effect or the importance of a result” (principle 5) — that is effect size. It is not the probability that the hypothesis is true or that the data arose by chance (principle 2). And “by itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis” (principle 6) — note by itself.

Which is why the third principle generalizes the point for anyone using tests to make decisions: conclusions “should not be based only on whether a p-value passes a specific threshold.” A test is a rule for controlling error rates across repeated use. It is a component of a decision, not the decision — and its error guarantees must cover the procedure actually used. A fixed-sample design uses its planned final analysis; a valid sequential design can instead allow a data-dependent sample size while controlling errors across interim looks and stopping. Predeclaring an otherwise invalid rule is insufficient. See statistically significant for how the verdict is usually reported and misreported.

References

• NIST/SEMATECH e-Handbook of Statistical Methods, §7.1.3 “What are statistical tests?”
• §7.1.4 “What are confidence intervals?”
• American Statistical Association, statement on statistical significance and p-values (2016).

For inference valid across monitoring times, see Howard et al., Time-uniform, nonparametric, nonasymptotic confidence sequences.