1. Definition

  • The significance level (α) is the threshold probability used in hypothesis testing to decide whether to reject the null hypothesis (H₀).
  • It represents the maximum risk of making a Type I error (rejecting H₀ when it is actually true) that you are willing to accept.

Common choices:

  • α = 0.05 (5%) → most widely used.
  • α = 0.01 (1%) → stricter, stronger evidence needed.
  • α = 0.10 (10%) → more lenient, weaker evidence acceptable.

2. Role in Hypothesis Testing

  • You calculate a test statistic (e.g., t, z, χ²) and get a p-value.
  • Compare p-value with α:
    • If p ≤ α → reject H₀ (evidence is strong enough).
    • If p > α → fail to reject H₀ (not enough evidence).

So α is your decision cutoff.

3. Type I Error and α

  • Type I Error = rejecting a true null hypothesis (false positive).
  • α is the probability of making a Type I Error.
  • Example: If α = 0.05, then in the long run, 5 out of 100 tests may incorrectly reject H₀ even though it is true.

4. Choosing α

  • 0.05 → balance between risk of false positives and sensitivity.
  • 0.01 → used in medicine, genetics, and high-stakes areas (very cautious).
  • 0.10 → used in exploratory research where missing possible effects is riskier than false alarms.

5. Example

Suppose you are testing whether a new website design improves conversion:

  • H₀: New design has no effect (conversion = 5%).
  • H₁: New design changes conversion rate.
  • Chosen α = 0.05.
  • You run an A/B test, compute p = 0.03.

Decision:

  • p = 0.03 ≤ 0.05 → reject H₀.
  • Conclude: statistically significant evidence that new design affects conversion.

6. Significance Level vs Confidence Level

  • Confidence level = $1 – α$.
  • Example: α = 0.05 → 95% confidence level.
  • Meaning: If you repeated the experiment many times, 95% of the confidence intervals would contain the true population parameter.

7. Misinterpretations to Avoid

  • α ≠ probability that H₀ is true.
  • α is chosen before the experiment, not after.
  • Statistical significance ≠ practical importance (need effect size too).

In short:
The significance level (α) is the threshold for deciding when to reject H₀, and it equals the risk of making a Type I error. Commonly α = 0.05, but stricter (0.01) or more lenient (0.10) levels are used depending on context.