Holdback Group
A randomized holdback can estimate a policy’s continuing effect when assignment, follow-up, treatment history, and the outcome window support the comparison.
-
-
Interpreting Analytical Results
The question is not whether a result is significant. It is what the sample represents, how wide the uncertainty is, and whether the difference matters.
-
Multiple Comparisons
Slicing a metric until something stands out is a search procedure. Selected findings need error control that accounts for the search.
-
Minimum Detectable Lift (MDL)
A 12% lift at target power does not rule out detecting 2%. State the power, effect units, and design before judging whether the experiment is worthwhile.
-
Sample size
Ten thousand events from two thousand users is not n = 10,000 for a question about users. The unit you sampled decides what n is.
-
A/B Testing
Randomization makes groups comparable in expectation. Estimate assignment effects with uncertainty and check departures from the intended design.
-
Sample Ratio Mismatch
A sample ratio mismatch is an alarm about experiment counts. It prompts investigation of assignment, exposure, logging, and filtering before outcome differences are trusted.