Optimizely
Twenty metrics, five true effects: Bonferroni finds 2.45 of five true effects, Benjamini-Hochberg 3.26.
-
-
Stopping Rules
When an experiment ends must be decided before the results are seen. The kinds of rule in use, which error rate each protects, and what looking early costs.
-
Sequential Probability Ratio Test (SPRT)
A fixed test needs 69 observations; the SPRT averages 34.3 under the null. Its 99.9th percentile is nine times its median.
-
Risk of Peeking
Daily monitoring need not change inference. Stopping on a fixed-sample p-value does; valid sequential methods account for repeated looks and stopping.
-
Sequential Testing
Looking at a running experiment repeatedly, legitimately. What naive peeking does to the error rate, and how planned boundaries hold it.
-
A/B Testing and Experiment Design
An offline metric argues a model might help; an experiment is the evidence it did.