1. What Is Statistical Significance?

Statistical significance refers to

whether an observed result is unlikely to have occurred by random chance alone, assuming the null hypothesis (H₀) is true.

A result is called statistically significant when:

  • the p-value ≤ α (predefined significance level)

2. The Role of Statistical Significance

Statistical significance answers one narrow question:

“Is this result sufficiently inconsistent with the null hypothesis?”

It does not answer:

  • How large the effect is
  • Whether the effect is important
  • Whether the result is correct or replicable

3. Relationship to Hypothesis Testing

Statistical significance arises within the hypothesis testing framework:

  1. Specify H₀ and H₁
  2. Choose α (e.g., 0.05)
  3. Compute a test statistic
  4. Compute the p-value
  5. Compare p-value with α

Decision:

  • p ≤ α → statistically significant
  • p > α → not statistically significant

4. What the p-value Really Means

p-value is

the probability, under H₀, of observing data as extreme or more extreme than what was observed.

Important clarifications:

  • p-value ≠ P(H₀ is true)
  • p-value ≠ probability the result is due to chance
  • p-value measures data extremeness, not truth

5. Statistical vs Practical Significance

Statistical Significance

  • Concerns detectability
  • Influenced by sample size and variability

Practical (or Clinical) Significance

  • Concerns real-world importance
  • Depends on context and effect size

Key distinction:

A result can be statistically significant but practically meaningless.

6. Sample Size and Statistical Significance

Sample size has a strong effect:

  • Large n → small standard error → easier to achieve significance
  • Small n → large uncertainty → harder to achieve significance

Thus:

  • With very large samples, tiny effects can become significant
  • With small samples, meaningful effects may not reach significance

7. Statistical Significance vs Power

  • Statistical significance is a binary outcome (yes/no)
  • Power is a probability describing detection capability

Connections:

  • High power → higher chance of achieving significance when effects exist
  • Low power → non-significant results are ambiguous

8. Common Misinterpretations

“Statistically significant means important”

False — importance requires effect size and context

“Not significant means no effect”

False — may reflect low power or high variability

“p = 0.05 is a magic threshold”

False — α is a convention, not a law of nature

9. Thresholds and Conventions

Common α levels:

  • 0.05 (most common)
  • 0.01 (more stringent)
  • 0.10 (exploratory contexts)

These thresholds reflect error tolerance, not evidence strength.

10. Statistical Significance as a Decision Tool

Statistical significance should be viewed as:

  • A decision rule, not a truth statement
  • A way to control false positives under repeated use

It is part of a risk-management system, not a proof system.

11. Reporting Beyond Significance

Good statistical practice includes:

  • Effect sizes
  • Confidence intervals
  • Sample size and power considerations
  • Domain-specific relevance

Statistical significance alone is insufficient.

12. Key Takeaway Statements

  • Statistical significance assesses inconsistency with H₀
  • It does not measure effect size or importance
  • It depends strongly on sample size
  • Non-significant ≠ no effect
  • Significance is a tool, not a conclusion

13. Concept Map

Data
 ↓
Test Statistic
 ↓
p-value
 ↓
Compare with α
 ↓
Statistically Significant or Not