Group Sequential Testing

Group sequential testing analyzes the data at several prespecified points and allows the study to stop at any of them, while keeping the overall false positive rate at the level you chose. It is the answer to the fact that fixed-horizon testing does not allow an unadjusted interim significance result to serve as an early efficacy declaration. Stopping for safety or operational faults is a different decision and may require an adjusted inferential analysis.

The mechanism is a budget. The 0.05 is spent across the planned looks rather than used once, so each look is judged against a stricter threshold than 0.05 and the total comes out right.

The budget, measured

import numpy as np

rng = np.random.default_rng(0)
N = 400_000

def looks(K, drift=0.0):
    """Cumulative z at K equally spaced looks. drift is the true effect,
       scaled so that drift=1 means the design's target effect at full data."""
    steps = rng.normal(size=(N, K)) / np.sqrt(K)
    B = np.cumsum(steps, axis=1) + drift * 2.8016 * np.arange(1, K + 1) / K
    return B / np.sqrt(np.arange(1, K + 1) / K)

POCOCK = {1: 1.960, 2: 2.178, 3: 2.289, 4: 2.361, 5: 2.413, 10: 2.555}
print(f"{'looks':>7}{'naive 1.96':>13}{'Pocock constant':>18}{'with that boundary':>21}")
for K, c in POCOCK.items():
    z = looks(K)
    naive = float((np.abs(z) >= 1.96).any(axis=1).mean())
    spent = float((np.abs(z) >= c).any(axis=1).mean())
    print(f"{K:>7}{naive:>13.4f}{c:>18.3f}{spent:>21.4f}")

print(f"\n{'design':>28}{'power':>9}{'average looks used':>21}{'of the full sample':>21}")
for name, bounds in (("fixed horizon, one test", [np.inf, np.inf, np.inf, 1.960]),
                     ("Pocock, 4 looks", [2.361] * 4),
                     ("O'Brien-Fleming, 4 looks", [4.048, 2.862, 2.337, 2.024])):
    z = looks(4, drift=1.0)
    crossed = np.abs(z) >= np.asarray(bounds)
    stopped = crossed.any(axis=1)
    first = np.where(stopped, crossed.argmax(axis=1) + 1, 4)
    print(f"{name:>28}{stopped.mean():>9.4f}{first.mean():>21.2f}{first.mean() / 4:>20.1%}")
#   looks   naive 1.96   Pocock constant   with that boundary
#       1       0.0503             1.960               0.0503
#       2       0.0828             2.178               0.0499
#       3       0.1075             2.289               0.0503
#       4       0.1254             2.361               0.0501
#       5       0.1410             2.413               0.0502
#      10       0.1927             2.555               0.0496
#
#                       design    power   average looks used   of the full sample
#      fixed horizon, one test   0.8003                 4.00              100.0%
#              Pocock, 4 looks   0.7192                 2.87               71.8%
#     O'Brien-Fleming, 4 looks   0.7917                 3.26               81.6%

The first table puts the problem and the fix side by side. Testing at 1.96 at every look gives 0.0828 with two looks, 0.1254 with four, and 0.1927 with ten. Replace 1.96 with the constant for that number of looks and every row returns to 0.05 — 0.0499, 0.0501, 0.0496.

Read the constants column rather than the result. The threshold rises with the number of looks you planned: 2.178 for two, 2.361 for four, 2.555 for ten. The design has to be declared in advance because the threshold depends on it. Deciding to add a fifth look after seeing the fourth is not a group sequential design; it is peeking with extra steps.

The general form of this is an alpha spending function, which says what fraction of the budget has been consumed by each fraction of the data. Two shapes dominate practice: Pocock uses a constant nominal boundary, not equal first-crossing alpha at each look; O'Brien–Fleming uses high early boundaries and much lower later ones. The Lan–DeMets formulation expresses either as a continuous function of information, which is what allows the looks to happen at times that were not exactly as planned.

What the saving actually costs

The second table is the part that is usually stated as a free lunch. At the effect the study was sized for, the fixed design reaches power 0.8003 using 100% of the data. Pocock uses 71.8% of it — and reaches power 0.7192. O'Brien–Fleming uses 81.6% and reaches 0.7917.

So at the same planned maximum, the expected saving is paid for in power: eight points for Pocock, one point for O'Brien–Fleming. The standard remedy is to raise the planned maximum a little — a few percent for O'Brien–Fleming, more for Pocock — so that power is preserved and the saving is still there in expectation. That inflation is the real price, and it means a group sequential design can end up using more data than the fixed design when the effect turns out to be small.

The honest summary is that these designs trade a worse worst case for a better average. Whether that is the right trade depends on what an extra week of traffic costs relative to the value of stopping a month early.

Efficacy, futility, and what to report

Boundaries come in two kinds and they behave differently. An efficacy boundary stops for a positive result and has to be paid for out of the alpha budget. A futility boundary stops when the effect is too small to be worth continuing; it costs power rather than alpha, and it is often the more valuable of the two in business settings, where most tested ideas do nothing.

One consequence is easy to miss when reporting. An estimate taken at the moment a boundary was crossed is biased away from zero, because crossing required an unusually large observed effect. Group sequential methods have adjusted estimators and confidence intervals for exactly this reason; quoting the raw effect from the stopping look overstates it.

Finally, the discipline these designs require is the same discipline fixed-horizon testing requires, applied to more decisions. The number of looks, their timing, the spending function, and the futility rule all belong in the plan written before the data. What group sequential testing removes is the need to choose between statistical validity and stopping early — not the need to decide in advance.

References

• NumPy documentation, numpy.random.Generator.normal. numpy.org
• SciPy documentation, scipy.stats.norm. docs.scipy.org
• gsDesign technical manual, Spending functions. Source