Statistical Significance, Practical Importance, and Testing Errors
Statistical significance and practical importance answer different questions.
A significance test evaluates whether the observed data are sufficiently inconsistent with a null hypothesis. It does not, by itself, determine whether the estimated difference is large enough to matter scientifically, medically, economically, or socially.
A Lead-Concentration Example
Suppose a reference or regulatory value for lead concentration is\[ 15\text{ parts per billion}. \]
Let\[ \mu \]
be the population mean lead concentration. We test\[ H_0:\mu=15 \]
against\[ H_A:\mu\neq15. \]
Suppose a large sample produces\[ \bar{x}=15.05. \]
The estimated difference from the reference value is\[ 15.05-15=0.05 \]
parts per billion.
This is a small numerical difference. Nevertheless, it can produce a small p-value if the estimate is sufficiently precise.
Test Statistics Compare Signal with Noise
A one-sample test statistic has the general form\[ \boxed{ \text{test statistic} = \frac{ \text{estimate}-\text{null value} }{ \text{standard error} } }. \]
For a population mean,\[ t = \frac{ \bar{x}-\mu_0 }{ s/\sqrt{n} }. \]
The numerator represents the observed effect:\[ \bar{x}-\mu_0. \]
The denominator represents sampling uncertainty:\[ \frac{s}{\sqrt{n}}. \]
A large test statistic can arise because:
- The estimated effect is large.
- The standard error is small.
- Both are true.
Therefore, a small p-value does not necessarily imply a large effect.
How a Tiny Difference Can Be Significant
Suppose\[ \bar{x}-\mu_0=0.05 \]
and the standard error is\[ SE=0.01. \]
Then\[ z = \frac{0.05}{0.01} = 5. \]
A standardized result five standard errors from the null value has an extremely small two-sided p-value.
The difference is statistically clear because it is large relative to its sampling uncertainty. However, its practical size remains only 0.05 parts per billion.
Statistical significance measures evidence against a null model. Practical importance concerns the magnitude and consequences of the estimated effect.
The Role of Sample Size
For an independent sample mean,\[ SE(\bar{X}) = \frac{\sigma}{\sqrt{n}}. \]
As \(n\) increases, the standard error decreases.
Suppose the true difference from the null value is a fixed nonzero quantity \(\delta\). The standardized statistic is approximately\[ z = \frac{\delta}{\sigma/\sqrt{n}} = \frac{\delta\sqrt{n}}{\sigma}. \]
As the sample size increases,\[ |z| \]
tends to increase, even though the effect \(\delta\) remains unchanged.
With a sufficiently large sample, a very small departure from the null hypothesis can become statistically significant.
Significance Does Not Measure Effect Size
A p-value does not report:
- The size of the difference
- The units of the difference
- The practical consequences
- Whether the effect exceeds a meaningful threshold
- Whether a policy should change
- Whether the study design is unbiased
- Whether the result will replicate
To assess magnitude, report an effect estimate such as:\[ \bar{x}-\mu_0, \]\[ \hat{p}_1-\hat{p}_2, \]\[ \frac{\mu_1-\mu_2}{\sigma}, \]
or an appropriate risk ratio, odds ratio, or regression coefficient.
Confidence Intervals Add Magnitude and Precision
Suppose the estimated mean lead concentration is\[ 15.05 \]
with a 95% confidence interval of\[ (15.02,15.08). \]
This interval excludes the null value of 15, so a corresponding two-sided 5% test rejects\[ H_0:\mu=15. \]
The interval also shows that the plausible effect size is small on the measurement scale:\[ 15.02-15=0.02 \]
to\[ 15.08-15=0.08. \]
Equivalently, the confidence interval for the difference from 15 is\[ \boxed{ (0.02,0.08) }. \]
The data support a positive difference, but one that appears numerically close to the reference value.
Regulatory Thresholds Require Context
A small numerical difference should not automatically be dismissed as practically irrelevant.
If 15 parts per billion is a legal, clinical, or safety threshold, even a small exceedance may trigger an important decision. Interpretation depends on:
- How the threshold was defined
- Measurement accuracy
- Individual versus population-average risk
- Consequences of exceeding the threshold
- Whether exposure effects are nonlinear
- Whether vulnerable populations are affected
- Costs of false reassurance and false alarms
- Whether the parameter of interest should be a mean, maximum, percentile, or exceedance rate
A statistically precise population mean near a health limit may be highly relevant even when the numerical difference is small.
Practical importance must be determined from subject-matter considerations, not from the p-value alone.
Confidence Intervals and Hypothesis Tests
There is a close relationship between two-sided confidence intervals and two-sided hypothesis tests.
For compatible procedures:
A two-sided test of \(H_0:\theta=\theta_0\) at significance level \(\alpha\) rejects exactly those null values \(\theta_0\) lying outside the corresponding \(100(1-\alpha)\%\) confidence interval.
For a 5% significance level,\[ \alpha=0.05, \]
the corresponding confidence level is\[ 1-\alpha=0.95. \]
Thus:
- If \(\theta_0\) is outside the 95% confidence interval, reject \(H_0\) at the 5% level.
- If \(\theta_0\) is inside the 95% confidence interval, fail to reject \(H_0\) at the 5% level.
Example of Test–Interval Duality
Suppose the 95% confidence interval for \(\mu\) is\[ (15.02,15.08). \]
Consider\[ H_0:\mu=15. \]
Because\[ 15\notin(15.02,15.08), \]
the corresponding two-sided test rejects \(H_0\) at the 5% level.
Now consider\[ H_0:\mu=15.04. \]
Because\[ 15.04\in(15.02,15.08), \]
the corresponding test fails to reject that null value at the 5% level.
The confidence interval can therefore be viewed as the collection of parameter values not rejected by the associated two-sided test.
The Procedures Must Match
The confidence-interval and testing conclusions align exactly only when they are constructed from compatible methods.
They should use the same:
- Statistical model
- Standard-error calculation
- Tail convention
- Significance and confidence levels
- Small-sample correction
- Variance assumptions
- Treatment of nuisance parameters
For example, an exact binomial test may not align perfectly with a Wald confidence interval. To obtain exact duality, the confidence set should be formed by inverting the exact test.
One-Sided Tests and Confidence Bounds
A one-sided test corresponds to a one-sided confidence bound, not necessarily to an ordinary two-sided interval at the same numerical confidence level.
For example, a one-sided test at\[ \alpha=0.05 \]
corresponds naturally to a 95% one-sided confidence bound.
It also has a relationship with a 90% two-sided interval under symmetric normal-theory procedures, because each tail of a 90% interval contains 5%.
The direction and tail convention must be reported explicitly.
Type I Error
A Type I error occurs when the null hypothesis is true but the test rejects it:\[ \boxed{ \text{Type I error} = \text{rejecting a true }H_0 }. \]
It is also called a false positive.
The significance level is defined as\[ \alpha = P(\text{reject }H_0\mid H_0\text{ true}). \]
If a test is calibrated at\[ \alpha=0.05, \]
its long-run Type I error probability is 5% or no greater than 5%, depending on the procedure.
Why “At Most 5%” Can Be Correct
For continuous test statistics, a properly calibrated test can often achieve\[ P(\text{reject }H_0\mid H_0)=0.05 \]
exactly.
For discrete data, such as binomial counts, it may be impossible to create a rejection region with exactly 5% probability. An exact nonrandomized test is then often conservative:\[ P(\text{reject }H_0\mid H_0)\leq0.05. \]
Therefore, “at most 5%” is especially appropriate for many exact discrete tests.
This control assumes:
- The null model is correct.
- The test was selected properly.
- The significance threshold was chosen in advance.
- All conducted analyses are accounted for.
- The study’s sampling assumptions hold.
Type II Error
A Type II error occurs when the null hypothesis is false but the test fails to reject it:\[ \boxed{ \text{Type II error} = \text{failing to reject a false }H_0 }. \]
It is also called a false negative.
The probability of a Type II error is denoted by\[ \beta. \]
Unlike \(\alpha\), \(\beta\) generally depends on the particular true alternative.
For example, the probability of missing a difference of 0.01 may be much larger than the probability of missing a difference of 2.0.
Power
Statistical power is the probability of correctly rejecting the null hypothesis under a specified alternative:\[ \boxed{ \text{Power}=1-\beta }. \]
Power increases when:
- The true effect is larger.
- The sample size is larger.
- Variability is smaller.
- Measurements are more reliable.
- The study design is more efficient.
- The significance level is larger.
- A justified directional test is used.
A test with low power may fail to detect a practically important effect.
The Decision Table
The relationship between the true state and the test decision can be summarized as follows:
| Reality | Reject \(H_0\) | Fail to reject \(H_0\) |
|---|---|---|
| \(H_0\) is true | Type I error | Correct decision |
| \(H_0\) is false | Correct rejection | Type II error |
The testing procedure balances the risks of these errors.
Reducing \(\alpha\) without increasing the sample size generally increases \(\beta\). Requiring stronger evidence against \(H_0\) reduces false positives but makes false negatives more likely.
The 5% Threshold Is a Convention
The rule\[ \text{reject }H_0 \quad\text{if}\quad p\leq0.05 \]
is common, but 5% is not a universal law.
An appropriate significance level depends on the consequences of errors.
Examples:
- A screening analysis may tolerate more false positives to avoid missing serious problems.
- A claim requiring strong evidence may use \(\alpha=0.01\).
- A safety procedure may prioritize avoiding false negatives.
- Multiple-testing settings may require a much smaller per-test threshold.
The choice should be justified by the application rather than adopted mechanically.
A P-Value Is Not the Type I Error Probability After Testing
After obtaining a p-value of 0.03, it is incorrect to say:
There is a 3% probability that the result is a false positive.
The p-value is\[ P(\text{data at least this inconsistent with }H_0\mid H_0). \]
It is not\[ P(H_0\text{ is true}\mid\text{data}). \]
The Type I error rate is a long-run property of a testing procedure under a true null hypothesis. It is not the posterior probability that an individual significant result is false.
Large Samples Can Detect Trivial Differences
Suppose\[ \delta=\mu-\mu_0 \]
is extremely small but nonzero. The standardized statistic is approximately\[ z = \frac{\delta\sqrt{n}}{\sigma}. \]
As\[ n\to\infty, \]
the magnitude of \(z\) increases without bound, provided \(\delta\neq0\).
Therefore, a test of an exact point null such as\[ H_0:\mu=15 \]
will eventually reject any fixed nonzero difference if the sample becomes sufficiently large.
This is mathematically correct, but it demonstrates why exact-zero null hypotheses can be less informative than questions about meaningful effect sizes.
Testing Practical Equivalence
Suppose differences smaller than\[ \Delta \]
are practically negligible.
Instead of testing exact equality, define an equivalence region:\[ -\Delta<\mu-\mu_0<\Delta. \]
An equivalence test asks whether the data provide sufficient evidence that the true difference lies inside this region.
One common method is the two one-sided tests procedure.
The hypotheses are structured so that practical equivalence must be demonstrated against both boundaries:\[ -\Delta \]
and\[ \Delta. \]
A confidence-interval interpretation is straightforward: an appropriately calibrated confidence interval must lie entirely within the equivalence region.
Testing Noninferiority
In some applications, the goal is not to prove two conditions are identical. It is to show that a new treatment is not worse than a reference by more than a meaningful amount \(\Delta\).
This is a noninferiority question.
For a difference where larger values are better, the hypotheses may be\[ H_0:\theta\leq-\Delta \]
versus\[ H_A:\theta>-\Delta. \]
Rejecting \(H_0\) supports the conclusion that the new treatment is not inferior beyond the prespecified margin.
The margin \(\Delta\) must be chosen using scientific or clinical reasoning—not selected after examining the data.
Confidence Intervals Communicate More Than Significance Labels
Consider two studies.
Study A
\[ \hat{\delta}=0.05, \qquad 95\%\ CI=(0.02,0.08). \]
This study provides strong evidence of a small positive difference.
Study B
\[ \hat{\delta}=3.0, \qquad 95\%\ CI=(-1.0,7.0). \]
This study has a much larger point estimate but greater uncertainty. It does not reject zero at the 5% level.
A binary significant–nonsignificant label conceals the important differences between these findings.
Multiple Testing Changes Error Rates
The 5% Type I error guarantee applies to one prespecified test. If many hypotheses are tested, the probability of at least one false positive can become much larger.
For \(m\) independent tests, each conducted at level \(\alpha\), the probability of at least one false rejection under all true nulls is\[ 1-(1-\alpha)^m. \]
For 20 tests at \(\alpha=0.05\),\[ 1-(0.95)^{20} \approx 0.642. \]
Thus, the chance of at least one false positive is approximately 64.2%.
Multiple-testing procedures may be needed to control:
- Family-wise error rate
- False discovery rate
- Other application-specific error criteria
A Better Reporting Strategy
Instead of reporting only whether \(p<0.05\), report:
- The estimated effect
- Its units
- A confidence interval
- The exact p-value
- The sample size
- The study design
- The practical threshold
- Important model assumptions
- Any multiple-testing adjustment
- Potential sources of bias
For the lead example, an informative report might say:
The estimated mean concentration was 15.05 parts per billion, 0.05 above the reference value. The 95% confidence interval for the mean was 15.02 to 15.08, and the corresponding interval for the difference from the reference was 0.02 to 0.08.
This conveys direction, magnitude, and precision without relying solely on a significance label.
Key Takeaway
Statistical significance and practical importance are not the same.
A test statistic compares an estimated effect with its standard error:\[ \text{test statistic} = \frac{ \text{estimate}-\text{null value} }{ SE }. \]
Because\[ SE\propto\frac{1}{\sqrt{n}}, \]
a very large sample can make a tiny effect statistically significant.
A confidence interval helps distinguish effect size from precision. For compatible two-sided procedures, a null value is rejected at the 5% level exactly when it lies outside the corresponding 95% confidence interval.
The two principal testing errors are:\[ \text{Type I error} = \text{rejecting a true }H_0 \]
and\[ \text{Type II error} = \text{failing to reject a false }H_0. \]
A 5% significance level controls the long-run Type I error rate at approximately—or, for conservative discrete tests, no more than—5% under the stated assumptions. It does not give the probability that a particular significant result is false. Reliable conclusions require effect estimates, confidence intervals, power considerations, subject-matter thresholds, and a valid study design.