Definition

A confidence interval (CI) is a range of values, derived from sample data, that is likely to contain the true population parameter (e.g., mean, proportion, regression coefficient) with a specified level of confidence.

Example: “We are 95% confident that the true mean lies between 4.8 and 5.2.”

Key Idea

• CI = point estimate ± margin of error
• The “confidence level” (usually 95%) means:
  • If you repeated the experiment many times, 95% of those CIs would contain the true value.
  • It does not mean “there is a 95% probability the true value is inside this interval” (common misconception).

Formula (for sample mean, large n)

$\text{CI} = \bar{x} \; \pm \; Z_{1-\alpha/2} \cdot \frac{s}{\sqrt{n}}$

Where:

• $\bar{x}$ = sample mean
• $s$ = sample standard deviation
• $n$ = sample size
• $Z_{1-\alpha/2}$​ = critical value (≈ 1.96 for 95% CI)

Example (Mean)

Suppose:

• Sample mean = 100
• Std. dev = 15
• Sample size = 50

Margin of error = $1.96 \times \frac{15}{\sqrt{50}} \approx 4.16$

So 95% CI = $100 \pm 4.16 = [95.84, 104.16]$

Interpretation: we are 95% confident the true mean lies between 95.8 and 104.2.

Example (Proportion)

• Out of 500 users, 120 converted → conversion rate = 24%.
• Standard error = $\sqrt{0.24 \cdot 0.76 / 500} \approx 0.019$.
• 95% CI = $0.24 \pm 1.96 \times 0.019 = [0.203, 0.277]$.

We are 95% confident the true conversion rate is between 20.3% and 27.7%.

Why Confidence Intervals Matter

• More informative than p-values → gives a range of plausible values, not just “significant / not significant.”
• Uncertainty quantification → wider CI = more uncertainty, narrower CI = more precision.
• Business communication → decision makers prefer “our uplift is between 2% and 5%” rather than “p < 0.05.”

Common Misinterpretations

Incorrect: “There’s a 95% chance the true value is inside this interval.”
Correct: “If we repeated this experiment many times, 95% of the intervals would cover the true value.”

In Python (with statsmodels)

import numpy as np
import statsmodels.stats.api as sms

data = np.array([100, 102, 98, 105, 97, 101, 99, 100])
ci = sms.DescrStatsW(data).tconfint_mean(alpha=0.05)
print("95% Confidence Interval:", ci)

import numpy as np
import statsmodels.stats.api as sms

data = np.array([100, 102, 98, 105, 97, 101, 99, 100])
ci = sms.DescrStatsW(data).tconfint_mean(alpha=0.05)
print("95% Confidence Interval:", ci)

Summary

• A confidence interval = range of plausible values for a population parameter.
• Constructed from sample data, includes uncertainty.
• Critical for A/B tests, regressions, forecasting, ML evaluation.