Parameter(s) of Interest
Ten segments with no effect anywhere: pick the best-looking one and the reported effect averages 0.0146.
-
-
Traditional A/B Test (Fixed-Horizon A/B Test)
A fixed-horizon A/B test needs a defined population, assignment, outcome, and stopping rule. Logging, dependence, and time variation affect interpretation.
-
Fixed-Horizon Testing
One test at the horizon gives 0.0497. Twenty unplanned looks give 0.2477 with the same statistic and the same 1.96.
-
Compromise Power Analysis
The power calculation for a sample you cannot change: fix the ratio of the two error rates, and let the critical value and both rates follow.
-
A/B/n Test
The winning arm is the maximum of several noisy estimates, so its measured lift is biased upward by construction.
-
Multivariate Test (MVT)
Testing several page elements in one experiment. What a factorial design buys, what it does not cost, and how the arm count multiplies the false positives.
-
Sequential Testing
Looking at a running experiment repeatedly, legitimately. What naive peeking does to the error rate, and how planned boundaries hold it.
-
Machine Learning Model Improvement Strategy
The hard part is not training a model. It is knowing which of ten possible improvements to try next.
-
A/B Testing and Experiment Design
An offline metric argues a model might help; an experiment is the evidence it did.
-
Pre-Registered Decision Gate
Write down what will be measured, what counts as enough, and what happens in each outcome before the pilot starts. The rule then limits how the result can be read.
-
Measuring an AI Development Tool Rollout
Dashboards can credit an AI coding tool for gaps that predate it. This guide covers pilot design, paired speed and quality measures, review cost, adoption, power, and gates.