Holdback Group
A randomized holdback can estimate a policy’s continuing effect when assignment, follow-up, treatment history, and the outcome window support the comparison.
-
-
Posterior probability of uplift
A direct answer to how likely it is that the treatment helped — and how much less comfortable that answer becomes once a threshold that matters is attached.
-
Conversion Rate Uplift
The change in conversion rate caused by a variant, on two scales that differ by a factor of the baseline. The relative figure needs its own interval formula.
-
Stopping Rules
When an experiment ends must be decided before the results are seen. The kinds of rule in use, which error rate each protects, and what looking early costs.
-
Bayesian Sequential Testing
One look at the end calls B better 5.03% of the time on identical variants. Looking every twenty users: 27.96%.
-
Traditional A/B Test (Fixed-Horizon A/B Test)
A fixed-horizon A/B test needs a defined population, assignment, outcome, and stopping rule. Logging, dependence, and time variation affect interpretation.
-
True Conversion Rate
The unobservable rate behind the number in the dashboard. What defines it, and why the textbook interval around it is not as safe as it looks.
-
Two-Proportion Z-Test
Do two groups convert at the same rate? The pooled z statistic, a worked A/B example with its interval, when to use Fisher’s exact test, and the A/B pitfalls.
-
Multivariate Test (MVT)
Testing several page elements in one experiment. What a factorial design buys, what it does not cost, and how the arm count multiplies the false positives.
-
Random Targeting Strategy
Random assignment supports the plain treated-versus-control comparison; random ranking supplies an uplift baseline. Observational estimates need additional identification and adjustment assumptions.
-
Conversion Rate (CR)
Conversions divided by something. Which something decides what the metric means, how it moves, and whether its error bars are honest.
-
Sequential Testing
Looking at a running experiment repeatedly, legitimately. What naive peeking does to the error rate, and how planned boundaries hold it.
-
A/B Testing
Randomization makes groups comparable in expectation. Estimate assignment effects with uncertainty and check departures from the intended design.
-
Incremental Conversions
The conversions a campaign caused, as opposed to the ones it was merely present for. What the scaling to a full population quietly assumes.
-
Offline Metrics and Online Outcomes
Offline scores measure predictions on logged data. Deployment value also depends on exposure, action capacity, outcome value, and intervention effects.
-
A/B Testing and Experiment Design
An offline metric argues a model might help; an experiment is the evidence it did.