Posterior probability of uplift

The posterior probability of uplift is the probability that a treatment helped, given the data observed. For a binary outcome with treated rate \(p_T\) and control rate \(p_C\), the quantity of interest is \(u = p_T - p_C\), and the number reported is \(\Pr(u > 0 \mid \text{data})\) — or, more usefully, \(\Pr(u > \tau \mid \text{data})\) for a threshold \(\tau\) that matters to the decision.

It answers the question people believe a p-value answers and does not. The American Statistical Association’s second principle is explicit that p-values “do not measure the probability that the studied hypothesis is true”. A posterior probability does, relative to the model and prior that produced it.

The Beta–Binomial calculation

With \(s_T\) successes out of \(n_T\) treated and \(s_C\) out of \(n_C\) controls, conditionally independent Binomial samples in the two groups, and independent Beta priors on their rates, the posteriors are conjugate and immediate:

\[ p_T \mid \text{data} \sim \text{Beta}(\alpha_T + s_T,\ \beta_T + n_T - s_T), \qquad p_C \mid \text{data} \sim \text{Beta}(\alpha_C + s_C,\ \beta_C + n_C - s_C) \]

The difference of two Beta variables has no convenient closed form, so the probability is obtained by sampling: draw from each posterior, subtract, and count the fraction of draws above the threshold. The same draws give the expected uplift and a credible interval from their quantiles, at no extra cost.

Two small points are worth settling once. Under continuous posteriors \(\Pr(u \ge 0)\) and \(\Pr(u > 0)\) differ by the probability of exactly zero, which is nil, so the distinction does not matter. A uniform \(\text{Beta}(1, 1)\) prior is a convenient proper choice for the rates, rather than a uniquely neutral description of ignorance. Its meaning depends on the parameterization; compare plausible baseline-informed or other priors when the data are sparse.

A worked example

The counts are the ones used under conversion rate uplift: 2,650 conversions from 50,000 treated visits against 2,500 from 50,000 control visits, a lift from 5.000 to 5.300 percent.

import numpy as np
from scipy import stats

def posterior_probability(s_t, n_t, s_c, n_c, threshold=0.0, draws=200_000, seed=3):
    """Beta-Binomial posteriors with uniform priors, compared by Monte Carlo."""
    rng = np.random.default_rng(seed)
    p_t = rng.beta(1 + s_t, 1 + n_t - s_t, draws)
    p_c = rng.beta(1 + s_c, 1 + n_c - s_c, draws)
    u = p_t - p_c
    return (u > threshold).mean(), u.mean(), np.quantile(u, [0.025, 0.975])

# 2,650 conversions from 50,000 against 2,500 from 50,000: a 0.3 point lift.
prob, expected, (low, high) = posterior_probability(2650, 50000, 2500, 50000)
print(f"{prob:.4f} {expected * 100:.4f} {low * 100:.4f} {high * 100:.4f}")
# 0.9842 0.3003 0.0269 0.5750

threshold = 0.002
# The same data, now against a business threshold of 0.2 percentage points.
prob_tau, _, _ = posterior_probability(2650, 50000, 2500, 50000, threshold=threshold)
print(f"{prob_tau:.4f}")
# 0.7631

p_t, p_c = 2650 / 50000, 2500 / 50000
# The frequentist two-sided p-value on the same counts, for comparison.
pooled = (2650 + 2500) / 100000
z = (p_t - p_c) / np.sqrt(pooled * (1 - pooled) * (1 / 50000 + 1 / 50000))
two_sided = 2 * (1 - stats.norm.cdf(abs(z)))
print(f"{z:.4f} {two_sided:.4f} {1 - two_sided / 2:.4f}")
# 2.1462 0.0319 0.9841

Question	Answer
Probability the treatment helped at all	0.9842
Expected uplift	+0.300 percentage points
95 percent credible interval	+0.027 to +0.575 points
Probability the uplift exceeds 0.2 points	0.7631

The last row is the one that changes decisions. A 98 percent chance that the treatment did something is comfortable; a 76 percent chance that it did enough to be worth the cost is a different conversation, and it is the conversation the business is actually having. Setting \(\tau\) to the smallest uplift that would justify shipping, rather than to zero, is most of the value of doing this at all.

It is not a different answer

The frequentist test on the same counts gives \(z = 2.1462\) and a two-sided p-value of 0.0319, so one minus half of it is 0.9841 — against a posterior probability of 0.9842. The agreement is not a coincidence: with flat priors and tens of thousands of observations the posterior for each rate is close to normal and centered on the observed proportion, so the Bayesian and frequentist computations are doing nearly the same arithmetic.

This is worth knowing in both directions. A posterior probability is not a way of extracting a stronger conclusion from the same data — anyone presenting it that way is misreading it. What it changes is what the number means: 0.98 is a statement about the uplift given the data, where 0.0319 is a statement about data given a null hypothesis, and only the first can be multiplied by a payoff. Where the two do diverge is with small samples, informative priors, or a threshold other than zero, which is where the method earns its place.

Beyond a binary outcome

• Continuous outcomes. For revenue or a similar measure, put Normal or Normal–Inverse-Gamma priors on the group means. The posterior of the difference \(d = \mu_T - \mu_C\) is approximately Normal, so \(\Pr(u > \tau) = 1 - \Phi\!\big((\tau - \hat{d})/\text{SE}_{\text{post}}\big)\) without sampling. The heavy tails that make incremental revenue hard to measure make this approximation worth checking rather than assuming.
• Per-segment uplift. A Bayesian logistic regression with treatment, covariates and their interactions gives posterior draws of the coefficients; for a segment \(x\), compute the uplift implied by each draw and take the fraction above \(\tau\). This is the Bayesian route to an uplift score that arrives with its own uncertainty.
• Many segments at once. Hierarchical priors pull small segments toward the overall effect, which can reduce exaggerated small-segment estimates when the pooling assumptions are appropriate, but does not guarantee correct claims. That shrinkage is the Bayesian counterpart of the variance penalty in causal trees.

What to fix before looking

The method makes two things easy that ought to be hard, and both need a decision recorded in advance.

The first is the threshold. \(\Pr(u > \tau) \ge q\) is a clean decision rule — ship if the probability of a worthwhile uplift is at least 90 percent, say — and it is only a rule if \(\tau\) and \(q\) were set before the data arrived. Choosing them afterward, when the posterior is already on screen, is choosing the conclusion.

The second is stopping. A posterior probability can be recomputed after every visitor, and watching it until it crosses a threshold inflates the chance of crossing it spuriously in the same way repeated frequentist testing does. Being Bayesian does not exempt the procedure from the consequences of an optional stopping rule chosen to favor a result; risk of peeking covers the mechanism. Monitor on a pre-committed schedule, or with a method designed to remain valid under continuous monitoring.

Finally, the probability is not yet the decision. A 90 percent chance of a small uplift and a 60 percent chance of a large one can favor opposite choices once the cost of treating is attached, which is the arithmetic under treatment cost. Reporting the probability, the expected uplift and the credible interval together — rather than any one of them — is what lets that comparison be made.

References

• American Statistical Association, “ASA Statement on Statistical Significance and P-Values”, 2016
• Ron Kohavi, Roger Longbotham, Dan Sommerfield and Randal M. Henne, “Controlled experiments on the web: survey and practical guide”, Data Mining and Knowledge Discovery 18(1), 2009
• SciPy documentation, scipy.stats.beta