Holdback Group

A holdback group is a randomly assigned share of eligible users, customers, or cases kept out of a specified feature, policy, or intervention for an extended period. Comparing outcomes by original assignment can estimate that policy’s effect over the defined period when follow-up is comparable and the design addresses spillovers, missingness, and implementation differences. A small percentage is often operationally convenient, but the required size depends on the effect and uncertainty that matter.

A launch experiment estimates an effect for its population and test period. It does not establish that the same effect will persist as customers, competitors, and the policy change. A continuing randomized comparison helps separate contemporaneous changes shared by both arms from the effect of assignment to the policy. Long-term experiments still face selection, attrition, and treatment-history problems. Other designs can answer some of these questions too, with different assumptions; a never-treated group is not the only possible source of causal evidence.

An example

Consider an illustrative subscription program. A launch experiment reports churn of 12% without offers and 9% under the offer policy in a high-risk subgroup defined before treatment. The team keeps a randomly assigned holdback for longer follow-up. Later, churn over a specified window is 14% in the holdback and 12% in the policy arm. These point estimates suggest a two-percentage-point reduction for that later comparison if assignment, observation, and interference assumptions hold. They do not alone prove that the market worsened or the effect fell from three points to two. That comparison needs uncertainty estimates and compatible populations, horizons, and policy definitions. Reselecting high-risk customers using scores altered by earlier offers can destroy comparability; use original assignment and a prespecified pretreatment subgroup, or explicitly design a different analysis.

Design choices

Keep assignment stable for the intended comparison, using a randomized allocation implemented with a suitable stable identifier and checking balance and assignment integrity. Define eligibility, treatment, outcome window, and the analysis of original assignments before reading results. Plan sample size for the minimum effect of interest, especially for rare outcomes. Excluding customers from a potentially useful policy has a cost that an authorized owner must accept, but a benefit is not guaranteed.

If prolonged exclusion is unsuitable, design a new comparison rather than routinely rotating customers and treating their histories as erased. Prior offers can have lasting effects. Re-randomization may estimate an effect of changing policy after that history; it does not automatically create a never-treated control. Assess carryover and whether any washout period is credible.

When a safeguard cannot be withheld, randomly delaying that same safeguard does not solve the constraint. A permissible comparison might keep the required baseline protection in both arms and randomize an additional change, or use a rollout design within the allowed timing. Observational comparisons and reviewed samples have their own targets and identification limits; reviewed samples can assess correctness without identifying the intervention’s causal business effect. Choose the design for the question and constraints rather than ranking every alternative below a holdback.

A holdback comparison concerns the whole policy as delivered, including the model, thresholds, and actions. It does not by itself identify which component caused a change, and its estimate applies to the population and period studied. Specify the intended outcome rather than relying only on a proxy metric, and report guardrail metrics alongside it. For revenue, distinguish receipts, discount costs, and net contribution without double counting. Holdback labels can inform outcomes without the specified intervention, but other policies, prior treatments, shared systems, or missing observations can still affect them. They are not automatically free of a feedback loop.

References

• Dmitriev et al. — Pitfalls of Long-Term Online Controlled Experiments (IEEE Big Data 2016), Microsoft Research publication abstract