Conversion Rate Uplift
The change in conversion rate caused by a variant, on two scales that differ by a factor of the baseline. The relative figure needs its own interval formula.
-
-
True Conversion Rate
The unobservable rate behind the number in the dashboard. What defines it, and why the textbook interval around it is not as safe as it looks.
-
Standard Error (SE)
Standard error describes sampling variability; a margin of error also uses a confidence level and reference distribution. A normal 95% interval uses about 1.96 times the SE.
-
True Mean (Population Mean, μ)
The population average an experiment is trying to learn: fixed, unobserved, and defined only once the population itself is.
-
Prediction Intervals (PI)
Interval width depends on the model and horizon. Coverage reflects the uncertainty represented in the model, not every possible change in the world.
-
Rule of Three
Zero failures do not imply a zero failure rate. The rule of three gives an approximate upper bound, while the exact formula shows how the approximation behaves at small sample sizes.
-
Statistical Inference for Machine Learning
Machine learning borrows its estimators from statistics and too often leaves the uncertainty behind.