1. Definition
- A Type I Error occurs when you reject the null hypothesis (H₀) even though it is actually true.
- In other words: a false positive.
- You conclude there is an effect/difference when in reality there isn’t.
2. Probability of Type I Error = α
- The significance level (α) is the probability of making a Type I error.
- Common choices:
- α = 0.05 → 5% chance of wrongly rejecting H₀.
- α = 0.01 → 1% chance.
- This is predefined before the test.
3. Example
Medical Example
- H₀: A new drug has no effect.
- H₁: The drug has an effect.
- If the drug truly has no effect, but your test (by chance) shows p < 0.05, you reject H₀.
- You conclude the drug works when it doesn’t → Type I Error.
A/B Testing Example
- H₀: Conversion rate A = Conversion rate B.
- H₁: Conversion rates differ.
- If in reality they are equal, but you find a “significant” difference (due to random variation), you’ve made a Type I Error.
4. Visual Intuition
Imagine two overlapping distributions (H₀ and H₁).
- The rejection region is set by α (e.g., the outer 5%).
- If your test statistic falls in that region while H₀ is true, you commit a Type I error.
5. Type I vs Type II Error
| Error Type | What Happens | Probability |
|---|---|---|
| Type I (False Positive) | Reject H₀ when it is true | α |
| Type II (False Negative) | Fail to reject H₀ when it is false | β |
| Power (1 – β) | Correctly reject H₀ when false | Depends on n, δ, α |
6. How to Control Type I Error
- Choose a stricter α (e.g., 0.01 instead of 0.05).
- Use Bonferroni correction or other multiple-testing corrections if running many tests.
- Ensure proper fixed-horizon testing (avoid peeking).
- Replicate experiments to confirm results.
In short:
A Type I Error is rejecting H₀ when it’s actually true → a false positive. Its probability is controlled by the significance level α (commonly 5%).