Type I Error

1. Definition

• A Type I Error occurs when you reject the null hypothesis (H₀) even though it is actually true.
• In other words: a false positive.
• You conclude there is an effect/difference when in reality there isn’t.

---

2. Probability of Type I Error = α

• The significance level (α) is the probability of making a Type I error.
• Common choices:
  • α = 0.05 → 5% chance of wrongly rejecting H₀.
  • α = 0.01 → 1% chance.
• This is predefined before the test.

---

3. Example

Medical Example

• H₀: A new drug has no effect.
• H₁: The drug has an effect.
• If the drug truly has no effect, but your test (by chance) shows p < 0.05, you reject H₀.
• You conclude the drug works when it doesn’t → Type I Error.

A/B Testing Example

• H₀: Conversion rate A = Conversion rate B.
• H₁: Conversion rates differ.
• If in reality they are equal, but you find a “significant” difference (due to random variation), you’ve made a Type I Error.

---

4. Visual Intuition

Imagine two overlapping distributions (H₀ and H₁).

• The rejection region is set by α (e.g., the outer 5%).
• If your test statistic falls in that region while H₀ is true, you commit a Type I error.

---

5. Type I vs Type II Error

Error Type	What Happens	Probability
Type I (False Positive)	Reject H₀ when it is true	α
Type II (False Negative)	Fail to reject H₀ when it is false	β
Power (1 – β)	Correctly reject H₀ when false	Depends on n, δ, α

---

6. How to Control Type I Error

• Choose a stricter α (e.g., 0.01 instead of 0.05).
• Use Bonferroni correction or other multiple-testing corrections if running many tests.
• Ensure proper fixed-horizon testing (avoid peeking).
• Replicate experiments to confirm results.

---

In short:
A Type I Error is rejecting H₀ when it’s actually true → a false positive. Its probability is controlled by the significance level α (commonly 5%).