Multiple Comparisons
Slicing a metric until something stands out is a search procedure. Selected findings need error control that accounts for the search.
-
-
Type I Error
You choose a nominal significance level. Actual Type I error and power depend on the test, assumptions, sample size, and effect.
-
Critical Value
The significance level translated onto the statistic’s scale. How 1.645 and 1.96 arise, why two-sided tests use alpha over two, and the mistakes it invites.
-
Statistical Significance
Statistical significance says how incompatible the data are with a null model at a chosen threshold. It does not say how large or how important an effect is.
-
Two-Proportion Z-Test
Do two groups convert at the same rate? The pooled z statistic, a worked A/B example with its interval, when to use Fisher’s exact test, and the A/B pitfalls.
-
Power (1 – β)
Power is the probability of rejecting the null when a specific alternative is true. Effect size, sample size, α, and variability set it, and it changes with the test design.
-
Hypothesis Testing
A test rejects or fails to reject — it never confirms. And “fail to reject” covers both no effect and not enough data to tell.
-
P-Value (probability value)
A p-value is a probability about data given a model. Reading it as a probability about the hypothesis inverts the conditional.
-
Z-Test
Estimate minus null value, over a known standard error, against the normal curve. Exact for normal data with known sigma, else approximate; mostly proportions.
-
T-Test
Estimate over its estimated standard error, referred to Student’s t. One-sample, paired, and two-sample forms, when to use Welch, and heavy tails.
-
Statistically Significant
“Unlikely due to chance” is the one reading a p-value cannot support. What it measures is incompatibility with a model, assumptions included.
-
Classifier Two-Sample Tests (C2STs)
A random forest scores 1.0000 in training on two samples drawn from the same distribution. Held out, it scores 0.4700.
-
Statistical Tests
Which test answers which question, from one-sample t to Fisher’s exact and Kolmogorov–Smirnov, how to choose by question, and the limits of a rejection.
-
Noninferiority Margin
A noninferiority test asks whether a new option is worse than the old one by no more than a margin chosen in advance, and it passes only when the confidence interval’s lower bound clears that margin.
-
McNemar Test
Compare two systems on the same cases by counting where only one passes. The exact test and worked counts explain why pairing changes the evidence for a difference.