Statistical Significance

1. What Is Statistical Significance?

Statistical significance refers to

whether an observed result is unlikely to have occurred by random chance alone, assuming the null hypothesis (H₀) is true.

A result is called statistically significant when:

• the p-value ≤ α (predefined significance level)

2. The Role of Statistical Significance

Statistical significance answers one narrow question:

“Is this result sufficiently inconsistent with the null hypothesis?”

It does not answer:

• How large the effect is
• Whether the effect is important
• Whether the result is correct or replicable

3. Relationship to Hypothesis Testing

Statistical significance arises within the hypothesis testing framework:

1. Specify H₀ and H₁
2. Choose α (e.g., 0.05)
3. Compute a test statistic
4. Compute the p-value
5. Compare p-value with α

Decision:

• p ≤ α → statistically significant
• p > α → not statistically significant

4. What the p-value Really Means

p-value is

the probability, under H₀, of observing data as extreme or more extreme than what was observed.

Important clarifications:

• p-value ≠ P(H₀ is true)
• p-value ≠ probability the result is due to chance
• p-value measures data extremeness, not truth

5. Statistical vs Practical Significance

Statistical Significance

• Concerns detectability
• Influenced by sample size and variability

Practical (or Clinical) Significance

• Concerns real-world importance
• Depends on context and effect size

Key distinction:

A result can be statistically significant but practically meaningless.

6. Sample Size and Statistical Significance

Sample size has a strong effect:

• Large n → small standard error → easier to achieve significance
• Small n → large uncertainty → harder to achieve significance

Thus:

• With very large samples, tiny effects can become significant
• With small samples, meaningful effects may not reach significance

7. Statistical Significance vs Power

• Statistical significance is a binary outcome (yes/no)
• Power is a probability describing detection capability

Connections:

• High power → higher chance of achieving significance when effects exist
• Low power → non-significant results are ambiguous

8. Common Misinterpretations

“Statistically significant means important”

False — importance requires effect size and context

“Not significant means no effect”

False — may reflect low power or high variability

“p = 0.05 is a magic threshold”

False — α is a convention, not a law of nature

9. Thresholds and Conventions

Common α levels:

• 0.05 (most common)
• 0.01 (more stringent)
• 0.10 (exploratory contexts)

These thresholds reflect error tolerance, not evidence strength.

10. Statistical Significance as a Decision Tool

Statistical significance should be viewed as:

• A decision rule, not a truth statement
• A way to control false positives under repeated use

It is part of a risk-management system, not a proof system.

11. Reporting Beyond Significance

Good statistical practice includes:

• Effect sizes
• Confidence intervals
• Sample size and power considerations
• Domain-specific relevance

Statistical significance alone is insufficient.

12. Key Takeaway Statements

• Statistical significance assesses inconsistency with H₀
• It does not measure effect size or importance
• It depends strongly on sample size
• Non-significant ≠ no effect
• Significance is a tool, not a conclusion

13. Concept Map

Data
 ↓
Test Statistic
 ↓
p-value
 ↓
Compare with α
 ↓
Statistically Significant or Not

Data
 ↓
Test Statistic
 ↓
p-value
 ↓
Compare with α
 ↓
Statistically Significant or Not