Posterior probability of uplift

Goal (uplift): $u = p_T – p_C$​ for a binary outcome (e.g., conversion).
Target quantity: $\Pr(u > 0 \mid \text{data})$ or, more generally, $\Pr(u > \tau \mid \text{data})$ for a business threshold $\tau$.

---

1) Simple and robust: Beta–Binomial model (binary outcomes)

Data.

• Treatment: $s_T$​ successes out of $n_T$
• Control: $s_C$​ successes out of $n_C$

Priors (conjugate).

• $p_T \sim \text{Beta}(\alpha_T,\beta_T)$, $p_C \sim \text{Beta}(\alpha_C,\beta_C)$
  (Common weak choice: $\alpha=\beta=1$, uniform)

Posteriors.

• $p_T \mid \text{data} \sim \text{Beta}(\alpha_T+s_T,\; \beta_T+n_T-s_T)$
• $p_C \mid \text{data} \sim \text{Beta}(\alpha_C+s_C,\; \beta_C+n_C-s_C)$

Posterior probability of uplift.
Because $u = p_T – p_C$​ has no closed form CDF here, compute by Monte Carlo:

1. Draw $p_T^{(m)} \sim \text{Beta}(\alpha_T+s_T, \beta_T+n_T-s_T)$
2. Draw $p_C^{(m)} \sim \text{Beta}(\alpha_C+s_C, \beta_C+n_C-s_C)$
3. Compute $u^{(m)} = p_T^{(m)} – p_C^{(m)}$
4. Approximate $\Pr(u>\tau \mid \text{data}) \approx \frac{1}{M}\sum_{m=1}^M \mathbf{1}\{u^{(m)}>\tau\}$

You can also report a credible interval for uuu from the empirical quantiles of $\{u^{(m)}\}$.

Boundary note: $\Pr(u \ge 0)$ and $\Pr(u>0)$ are practically identical under continuous posteriors; the difference at exactly 0 has probability ~0.

---

2) Continuous outcomes (e.g., revenue), Normal model

If outcomes $Y$ are approximately Normal with unknown means $\mu_T,\mu_C$ and (optionally) shared variance, place Normal–Inverse-Gamma (or Normal with known $\sigma^2$) priors. The posterior of the mean difference $d=\mu_T-\mu_C$​ is (approximately) Normal, so:

$\Pr(u> \tau \mid \text{data}) = \Pr(d > \tau \mid \text{data}) = 1 – \Phi\!\left(\frac{\tau – \hat{d}}{\text{SE}_{\text{post}}}\right),$

where $\hat{d}$ and $\text{SE}_{\text{post}}$ come from the posterior.

---

3) Bayesian logistic (or hierarchical) regression

With covariates $x$ and treatment indicator $T$:

$\Pr(Y=1\mid x,T)=\text{logit}^{-1}\!\big(\beta_0+\beta^\top x + \gamma T + \delta^\top(x\cdot T)\big).$

• Draw posterior samples of coefficients.
• For a target segment $x$, compute per-draw uplift $u^{(m)}(x) = \Pr(Y=1\mid x,T{=}1,\theta^{(m)}) – \Pr(Y=1\mid x,T{=}0,\theta^{(m)}),$ then estimate $\Pr(u(x) > \tau)$ by the fraction of draws exceeding $\tau$.
• Hierarchical priors help stabilize small segments.

---

4) What to report (decision-friendly)

• Posterior probability: $\Pr(u>0 \mid \text{data})$ and/or $\Pr(u>\tau \mid \text{data})$
• Expected uplift: $\mathbb{E}[u \mid \text{data}]$
• 95% credible interval for $u$
• Risk-aware decision rule: “Ship if $\Pr(u>\tau) \ge q$” (e.g., $q=0.9$)

---

5) Practical tips

• Use weakly informative priors (e.g., Beta(1,1) or Beta(0.5,0.5)) to avoid brittle results in small samples.
• Prefer $\Pr(u>\tau)$ over just point estimates when the cost of a bad launch is high.
• For many arms/segments, apply hierarchical modeling to borrow strength and reduce false positives.
• Avoid continual naive peeking; if monitoring, use always-valid Bayesian updates consistently.