Significance Level (α)

1. Definition

• The significance level (α) is the threshold probability used in hypothesis testing to decide whether to reject the null hypothesis (H₀).
• It represents the maximum risk of making a Type I error (rejecting H₀ when it is actually true) that you are willing to accept.

Common choices:

• α = 0.05 (5%) → most widely used.
• α = 0.01 (1%) → stricter, stronger evidence needed.
• α = 0.10 (10%) → more lenient, weaker evidence acceptable.

---

2. Role in Hypothesis Testing

• You calculate a test statistic (e.g., t, z, χ²) and get a p-value.
• Compare p-value with α:
  • If p ≤ α → reject H₀ (evidence is strong enough).
  • If p > α → fail to reject H₀ (not enough evidence).

So α is your decision cutoff.

---

3. Type I Error and α

• Type I Error = rejecting a true null hypothesis (false positive).
• α is the probability of making a Type I Error.
• Example: If α = 0.05, then in the long run, 5 out of 100 tests may incorrectly reject H₀ even though it is true.

---

4. Choosing α

• 0.05 → balance between risk of false positives and sensitivity.
• 0.01 → used in medicine, genetics, and high-stakes areas (very cautious).
• 0.10 → used in exploratory research where missing possible effects is riskier than false alarms.

---

5. Example

Suppose you are testing whether a new website design improves conversion:

• H₀: New design has no effect (conversion = 5%).
• H₁: New design changes conversion rate.
• Chosen α = 0.05.
• You run an A/B test, compute p = 0.03.

Decision:

• p = 0.03 ≤ 0.05 → reject H₀.
• Conclude: statistically significant evidence that new design affects conversion.

---

6. Significance Level vs Confidence Level

• Confidence level = $1 – α$.
• Example: α = 0.05 → 95% confidence level.
• Meaning: If you repeated the experiment many times, 95% of the confidence intervals would contain the true population parameter.

---

7. Misinterpretations to Avoid

• α ≠ probability that H₀ is true.
• α is chosen before the experiment, not after.
• Statistical significance ≠ practical importance (need effect size too).

---

In short:
The significance level (α) is the threshold for deciding when to reject H₀, and it equals the risk of making a Type I error. Commonly α = 0.05, but stricter (0.01) or more lenient (0.10) levels are used depending on context.