Sample Standard Deviation
Spread around the sample mean, dividing by n minus 1. Not the average deviation, not the standard error, and the sigma a sample-size calculation needs.
-
-
A Priori Power Analysis
Before collecting data, a priori power analysis supports design decisions under stated effect and variability assumptions. With a fixed sample, sensitivity calculations answer a different planning question.
-
Minimum Detectable Lift (MDL)
A 12% lift at target power does not rule out detecting 2%. State the power, effect units, and design before judging whether the experiment is worthwhile.
-
Trivial Effects
Significance says a difference is detectable. Whether it is worth anything is a separate question that no test answers.
-
Sample size
Ten thousand events from two thousand users is not n = 10,000 for a question about users. The unit you sampled decides what n is.
-
Power (1 – β)
Power is the probability of rejecting the null when a specific alternative is true. Effect size, sample size, α, and variability set it, and it changes with the test design.
-
Statistical Power
Power is not a property of a test but of a test plus an effect size you name. Without naming one, “underpowered” has no meaning.
-
Power Analysis
Sample size, effect size, α and power are four numbers with one equation. Fix any three and the fourth follows — which is the whole method.