Statistical Tests
1) Definition
- A statistical test is a formal procedure to decide whether observed data provides enough evidence to reject a null hypothesis (H₀) in favor of an alternative hypothesis (H₁).
- Based on probability theory and sampling distributions.
In plain terms: it’s a way to check “is this effect real, or just random chance?”
2) Key Ingredients
- Null hypothesis (H₀): default assumption (e.g., “no difference between groups”).
- Alternative hypothesis (H₁): what you want to test (e.g., “group A has higher mean than group B”).
- Test statistic: computed number summarizing evidence (e.g., t-value, χ²).
- p-value: probability of observing data this extreme (or more) if H₀ were true.
- Significance level (α): threshold (commonly 0.05) to decide whether to reject H₀.
3) Common Statistical Tests
a) Comparing means
- t-test:
- One-sample: compare mean vs known value.
- Two-sample: compare means of two groups.
- Paired: compare before/after in the same group.
- ANOVA: compare means across ≥3 groups.
b) Comparing distributions
- Chi-square test: test independence in contingency tables.
- Kolmogorov–Smirnov (KS) test: compare continuous distributions.
- Mann–Whitney U test: non-parametric test for median differences.
c) Comparing variances
- F-test / Levene’s test: test if groups have equal variance.
d) Correlation / association
- Pearson correlation test (linear correlation).
- Spearman rank test (monotonic correlation).
e) Drift detection in ML
- KS test → feature distribution shift.
- Chi-square → categorical drift.
- PSI (Population Stability Index) → shift in score distributions.
4) Example (t-test)
Suppose we A/B test a new website:
- Group A mean CTR = 0.12, Group B mean CTR = 0.15.
- t-test → p = 0.03 < 0.05 → reject H₀.
- Interpretation: the difference is statistically significant.
5) Limitations
- p-value ≠ effect size: significance doesn’t mean practical importance.
- Assumptions (normality, independence, variance equality) may not hold.
- Multiple testing → increases false positives (need corrections like Bonferroni, FDR).
- Sensitive to sample size (large n makes tiny effects significant).
6) Alternatives / Complements
- Confidence intervals: show effect size + uncertainty.
- Bayesian hypothesis testing: provides posterior probability of hypotheses.
- Effect size measures: Cohen’s d, odds ratios.
Summary
- Statistical tests = tools to decide if observed patterns are due to chance or real effects.
- Involve H₀, H₁, test statistic, p-value, α.
- Common types: t-tests, ANOVA, chi-square, KS test, Mann–Whitney, correlation tests.
- Widely used in A/B testing, drift detection, feature analysis, medical trials.