Statistical Significance

A result is statistically significant when its p-value is no larger than a significance level \(\alpha\) that was fixed before the data were examined. NIST/SEMATECH e-Handbook §7.1.3 describes \(\alpha\) as the risk of rejecting the null hypothesis when it is in fact true, often called the significance level of the test. Wasserstein and Lazar (2016), in the American Statistical Association’s statement on p-values, state that a p-value can indicate how incompatible the data are with a specified statistical model. Taken together, “significant” means incompatible with the null model at the chosen level, and nothing more.

The rule

1. State a null hypothesis \(H_0\) and an alternative.
2. Choose \(\alpha\) before looking at the data.
3. Compute the test statistic and its p-value.
4. If \(p \le \alpha\), the result is statistically significant at that level. Otherwise it is not.

Statistically Significant takes the wording of this rule apart, P-Value explains the number being compared, and Significance Level (α) covers the threshold.

It moves with the sample size

The numbers here are hypothetical. Two variants have conversion rates of 10.0% and 10.2%, a gap of 0.2 percentage points. A two-proportion z-test with a pooled variance gives these results as the number of users in each arm grows, and the gap stays the same.

The effect did not change. Only the amount of evidence did. At a million users per arm the same small gap is significant at \(\alpha = 0.05\), and it may still be too small to matter. The ASA statement puts it directly: statistical significance is not equivalent to scientific, human, or economic significance.

What it does not tell you

• It is not the probability that the null hypothesis is true or that chance alone produced the data. The ASA statement says a p-value is neither.
• It does not measure the size of an effect or the importance of a result.
• A decision should not rest only on whether a p-value passes a threshold.
• A result that is not significant is not evidence of no effect. A study with low power can miss a real one; see Statistical Power.

What to report beside it

Report the effect size and an interval for it (Effect Size (δ), Confidence Intervals), the exact p-value, and how many analyses were run, because the ASA statement notes that valid inference needs to know how many and which analyses were conducted and how results were selected for reporting. Practical Significance asks whether the effect matters.

References:

• NIST/SEMATECH e-Handbook of Statistical Methods, §7.1.3, What are statistical tests?
• Wasserstein and Lazar, The ASA’s Statement on p-Values: Context, Process, and Purpose, The American Statistician 70(2), 2016 (accepted manuscript)