Statistically Significant

A result is statistically significant if data at least this extreme would be unlikely under the null model — the null hypothesis together with the test’s assumptions — judged against a significance level chosen in advance.

The awkwardness of that sentence is deliberate. Every shorter version in circulation says something the test cannot support, and the American Statistical Association’s 2016 statement on p-values is explicit about which readings are wrong. Its six principles are the backbone of this entry.

What the p-value is, in the ASA’s words

The first principle states what a p-value can do: “P-values can indicate how incompatible the data are with a specified statistical model.”

Note the object. Not “with the hypothesis” but with a model — and a statistical model includes more than the absence of an effect. It carries the test’s assumptions about sampling, independence, and distribution. So a small p-value is evidence against the package as a whole, which is why a violated assumption can produce one just as a real effect can. A clustered sample analyzed as if it were independent will hand you significance that reflects the analysis rather than the world.

The second principle rules out the two readings people actually reach for: “P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.”

Both failures invert the conditional. The p-value is computed assuming the null model holds; it is a probability about data given a hypothesis, not about a hypothesis given data. Getting from one to the other requires something the test never supplies — a prior over hypotheses.

Which makes “unlikely due to chance” the phrase to retire. It compresses the definition into something that invites the reading “there is only a 5% chance this is a fluke,” and that sentence is exactly what the second principle says a p-value cannot support. It is also the most common summary in dashboards and experiment tools, so it is worth correcting where you find it. See p-value.

The machinery

Significance is a verdict produced by a procedure, so the pieces are worth naming precisely.

• Null hypothesis (H₀) — the assumption of no effect or no difference. “The new drug has no effect compared to placebo.”
• Alternative hypothesis (H₁) — that there is one. “The drug lowers blood pressure more than placebo.”
• p-value — the probability of a result at least as extreme as the observed one, computed under the null model, assumptions included.
• Significance level (α) — a threshold fixed in advance, commonly 0.05. See significance level.

The rule is then mechanical: p ≤ α, reject the null; p > α, fail to reject it. Two things about that rule are easy to lose. “Fail to reject” is not “accept” — it is the absence of a verdict, not a verdict of no effect. And α is a tolerance for false positives you are choosing, which is why it belongs to the design of the test rather than to its results: see Type I error and hypothesis testing.

Conventional levels — 0.05, 0.01 for stricter settings, 0.10 in exploratory work — express error tolerance, not strength of evidence. There is nothing in nature at 0.05, and the ASA’s third principle generalizes the point: “scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.”

What significance does not tell you

Three separate things, each a common failure.

How big the effect is. The fifth principle: “a p-value, or statistical significance, does not measure the size of an effect or the importance of a result.” Significance and magnitude are different quantities — see effect size.

Whether it matters. With a large enough sample almost any nonzero difference clears any threshold, because significance depends on the sample size while the effect does not. A 0.5% improvement measured across ten million users can be significant and not worth the engineering; so the question “is this big enough to act on?” has to be asked separately, against a threshold your domain sets. See sample size.

That there is no effect, when the result is non-significant. A non-significant result is consistent with no effect and also with an effect the study was too small to detect, and nothing in the p-value distinguishes them. The way to tell is to ask what the study could have detected — see statistical power. An underpowered test that returns “not significant” has told you about the test, not about the world.

The sixth principle draws the general conclusion: “by itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.” Not is a bad measure — by itself. The number is informative in company and misleading alone.

What to report with it

The ASA names the alternatives statisticians use to supplement or replace p-values: methods “that emphasize estimation over testing such as confidence, credibility, or prediction intervals; Bayesian methods; alternative measures of evidence such as likelihood ratios or Bayes factors; and other approaches such as decision-theoretic modeling and false discovery rates.”

The common thread in the first group is reporting a range for the quantity you care about rather than a verdict about a hypothesis — see confidence intervals and prediction intervals. An interval answers “how big, and how precisely do we know?” in one object, which is most of what a decision needs.

The fourth principle covers the other half: “proper inference requires full reporting and transparency.” This is the one that makes selective reporting a statistical problem rather than an ethical footnote. Run twenty comparisons at α = 0.05 and report the one that cleared it, and the reported p-value no longer means what it says — the ASA names the practice, noting “‘p-hacking’ and ‘data dredging’ that emphasize the search for small p-values over other statistical and scientific reasoning.” The defense is procedural: fix the metric and the analysis before looking, and report what you ran rather than what worked.

Which is the honest summary of the term. Statistical significance is a decision rule for controlling false positives under repeated use — part of a risk-management system rather than a proof system. It says something narrow and worth knowing: under this model, data like this would be uncommon. Everything else a reader wants — how large, how certain, whether to act — comes from other numbers reported beside it. See A/B testing for where this shows up most often in practice.

Further reading

• American Statistical Association, “ASA Releases Statement on Statistical Significance and P-Values”