Bayesian Stopping Rules

A Bayesian stopping rule is a criterion, fixed before the data, for when to stop collecting it — stated in terms of a posterior probability or a Bayes factor rather than a p-value.

The two common forms are a posterior threshold (“stop when \(P(H_1 \mid D) > 0.95\)”) and a Bayes factor threshold (“stop when \(BF > 10\)”). Both are rules, and a rule has properties the number inside it does not.

A threshold is not an error rate

import numpy as np
from scipy.stats import norm

RUNS, MAX_N, EVERY = 20_000, 4_000, 100

def run(p_a, p_b, hi, lo=None, seed=0):
    """Stop for success when P(B > A) exceeds hi, for futility when it falls
       below lo. Returns the rate of each outcome and where runs stopped."""
    rng = np.random.default_rng(seed)
    ca = np.cumsum(rng.random((RUNS, MAX_N)) < p_a, axis=1)
    cb = np.cumsum(rng.random((RUNS, MAX_N)) < p_b, axis=1)
    looks = np.arange(EVERY, MAX_N + 1, EVERY)
    n = looks[None, :]
    ka, kb = ca[:, looks - 1], cb[:, looks - 1]
    mean = lambda k: (1 + k) / (2 + n)
    var = lambda k: (1 + k) * (1 + n - k) / ((2 + n) ** 2 * (3 + n))
    prob = norm.cdf((mean(kb) - mean(ka)) / np.sqrt(var(ka) + var(kb)))
    win = prob > hi
    give_up = prob < lo if lo is not None else np.zeros_like(win)
    done = win | give_up
    first = np.where(done.any(axis=1), done.argmax(axis=1), len(looks) - 1)
    idx = (np.arange(RUNS), first)
    return float(win[idx].mean()), float(give_up[idx].mean()), looks[first]

print(f"  a success threshold on P(B > A), checked every {EVERY} users up to {MAX_N:,}:")
print(f"{'threshold':>11}{'wrong calls (no effect)':>26}{'right calls (2pp lift)':>25}"
      f"{'median n, lift':>17}")
for hi in (0.90, 0.95, 0.99, 0.999):
    bad, _, _ = run(0.10, 0.10, hi)
    good, _, stop = run(0.10, 0.12, hi)
    print(f"{hi:>11}{bad:>26.4f}{good:>25.4f}{np.median(stop):>17.0f}")
"""A posterior threshold is a decision rule. Its long-run error rate is a
   property of the whole rule, not of the number in it."""

print(f"{'success / futility':>21}{'wins (2pp lift)':>18}{'abandons':>11}{'median n':>11}")
for hi, lo in ((0.95, None), (0.95, 0.20), (0.95, 0.40)):
    good, quit_, stop = run(0.10, 0.12, hi, lo)
    label = f"{hi} / none" if lo is None else f"{hi} / {lo}"
    print(f"{label:>21}{good:>18.4f}{quit_:>11.4f}{np.median(stop):>11.0f}")
for hi, lo in ((0.95, None), (0.95, 0.20), (0.95, 0.40)):
    good, quit_, stop = run(0.10, 0.10, hi, lo)
    label = f"{hi} / none" if lo is None else f"{hi} / {lo}"
    print(f"    no effect, {label:>13}: wrongly wins {good:.4f}, abandons {quit_:.4f},"
          f" median n {np.median(stop):>5.0f}")

for bf, label in ((3, "substantial"), (10, "strong"), (30, "very strong"), (100, "decisive")):
    print(f"    BF = {bf:>3}: {label:<12} -> posterior probability "
          f"{bf / (1 + bf):.4f} at even prior odds")
#   a success threshold on P(B > A), checked every 100 users up to 4,000:
#   threshold   wrong calls (no effect)   right calls (2pp lift)   median n, lift
#         0.9                    0.4027                   0.9796              500
#        0.95                    0.2487                   0.9466              800
#        0.99                    0.0664                   0.8061             2000
#       0.999                    0.0106                   0.5124             4000
#    success / futility   wins (2pp lift)   abandons   median n
#           0.95 / none            0.9466     0.0000        800
#            0.95 / 0.2            0.8001     0.1741        500
#            0.95 / 0.4            0.6122     0.3788        300
#     no effect,   0.95 / none: wrongly wins 0.2487, abandons 0.0000, median n  4000
#     no effect,    0.95 / 0.2: wrongly wins 0.2150, abandons 0.5951, median n   500
#     no effect,    0.95 / 0.4: wrongly wins 0.1729, abandons 0.7893, median n   200
#     BF =   3: substantial  -> posterior probability 0.7500 at even prior odds
#     BF =  10: strong       -> posterior probability 0.9091 at even prior odds
#     BF =  30: very strong  -> posterior probability 0.9677 at even prior odds
#     BF = 100: decisive     -> posterior probability 0.9901 at even prior odds

The first table runs four success thresholds, checked every 100 users up to 4,000. Under no effect, a 0.95 rule wrongly declares a winner 24.87% of the time; 0.99 gives 6.64%; 0.999 gives 1.06%. None of these equals one minus the threshold, because the rule gets forty chances to cross.

The long-run error rate belongs to the full rule: prior, threshold, look schedule and maximum sample. Here posterior comparisons use a normal approximation to Beta differences. Simulation measures that implemented rule; exact enumeration, recursive calculation or suitable analytic bounds can also determine or bound its error. Ignorable stopping preserves exact posterior updating under the model, not a nominal frequentist error rate for every posterior threshold.

The same table prices the other side. Raising the threshold from 0.95 to 0.999 cuts wrong calls from 24.87% to 1.06% and cuts right calls on a real 2-point lift from 94.66% to 51.24%, while the median sample goes from 800 to the 4,000 cap. A threshold is a dial between those three quantities and nothing else.

Futility is usually the valuable half

The second block adds a rule for abandoning. On a real lift, a futility threshold at 0.20 drops the win rate from 94.66% to 80.01% and the median sample from 800 to 500 — a real cost in power.

On data with no effect the same rule abandons 59.51% of runs, while the median sample across all runs is 500 users, where without it the median run consumed the entire 4,000. Tightening futility to 0.40 abandons 78.93%, with a median sample of 200 across all runs.

That asymmetry is the argument for futility rules in product experimentation, where most tested ideas do nothing. The cost falls on the minority of experiments that would have won; the saving falls on the majority that would not. Worth noting too: futility also lowers the wrong-call rate slightly, from 24.87% to 21.50%, because runs are cut off before luck can carry them across.

Reading a Bayes factor threshold

The last block illustrates posterior probabilities at selected Bayes factor values: \(BF = 3\) is 0.7500, \(BF = 10\) is 0.9091, \(BF = 100\) is 0.9901 — at even prior odds, and only there. At prior odds of 1 to 9, a Bayes factor of 10 leaves the posterior probability at 10/19, approximately 0.5263.

The bands themselves are a convention with no more standing than 0.05 has. “Strong evidence” is a label someone chose, and where a decision is at stake the threshold should come from the cost of each mistake rather than from the scale — which is what Bayesian decision theory makes explicit.

One further caution specific to Bayes factors: they depend on the prior far more sharply than a posterior does, because a marginal likelihood is an average over the prior rather than something data eventually overwhelms. A stopping rule built on a Bayes factor inherits that sensitivity along with everything else. For how the rules behave in a running experiment, see Bayesian sequential testing.

References

• SciPy documentation, scipy.stats.norm. docs.scipy.org