Pre-Registered Decision Gate

A pre-registered decision gate is a decision rule written down and saved before a pilot or experiment produces data. It names the primary metric, the smallest effect worth acting on, the guardrail metrics and the harm each may cause, the interval or test the analysis will use, and which decision follows from each combination of results. When the data arrive, the analysis applies the rule and reports what it returns.

What the gate fixes

• The primary metric. One measure decides the effect question, so that a good result on some other measure cannot be substituted for it.
• A worthwhile threshold. The smallest effect that justifies acting, chosen by the people who bear the cost, not by what the data happened to show.
• Guardrail margins. How much harm on secondary measures, such as rework or review wait, is tolerable.
• The analysis. The comparison, the level of the interval, the unit of analysis, and how missing data and exclusions are handled.
• The decision for each outcome. Including the uncomfortable one: what happens if the evidence is inconclusive.

A rule can be as small as a configuration file and a short function that reads the estimates and returns EXPAND, EXTEND, or STOP, as long as it is saved before the analysis runs and used unchanged. An extension plan should also fix the added sample size, the next analysis time, and how repeated testing will be handled; repeatedly applying a fixed-sample interval to growing data changes its error rate. Having an EXTEND outcome matters: without it, an inconclusive pilot is pushed into a yes or a no by whoever reads the dashboard.

What goes wrong without it

Choosing the metric, the threshold, or the subgroup after seeing the results turns the analysis into a search, and a search finds something. The simulation below has no true effect on any of five metrics. A pilot compares 15 treated teams with 15 control teams and expands if the lower end of a 90 percent interval for a metric is above zero. In one version the metric is named in advance; in the other, the best of the five is chosen afterward. Python 3.12.14, numpy 2.2.3, scipy; 5,000 simulated pilots.

import numpy as np
from scipy import stats
rng = np.random.default_rng(3)
def lower90(a, b):
    d = a.mean() - b.mean()
    se = np.sqrt(a.var(ddof=1)/len(a) + b.var(ddof=1)/len(b))
    df = (a.var(ddof=1)/len(a) + b.var(ddof=1)/len(b))**2 / ((a.var(ddof=1)/len(a))**2/(len(a)-1) + (b.var(ddof=1)/len(b))**2/(len(b)-1))
    return d - stats.t.ppf(0.95, df) * se
reps, k, teams = 5000, 5, 15
fixed = best = 0
for _ in range(reps):
    lows = [lower90(rng.normal(0, 1, teams), rng.normal(0, 1, teams)) for _ in range(k)]   # no true effect on any of k metrics
    fixed += lows[0] > 0           # metric named before the data
    best  += max(lows) > 0         # metric chosen after seeing all five
print(f"no true effect on any metric, 15 vs 15 teams, expand if the one-sided 95% lower bound > 0")
print(f"  primary metric fixed in advance : expands in {fixed/reps:.1%} of pilots")
print(f"  best of {k} metrics chosen later  : expands in {best/reps:.1%} of pilots")

no true effect on any metric, 15 vs 15 teams, expand if the one-sided 95% lower bound > 0
  primary metric fixed in advance : expands in 5.3% of pilots
  best of 5 metrics chosen later  : expands in 22.3% of pilots

A metric fixed in advance expands 5.3 percent of the time, which is what a one-sided 95 percent bound should give when nothing is happening. Picking the best of five afterward expands in 22.3 percent of pilots, close to the \(1-0.95^5 \approx 22.6\) percent expected if the five metrics were independent. The data were the same; the procedure differs. The same inflation arises from choosing the subgroup, the time window, or the threshold after the fact, and from looking repeatedly while the pilot runs and stopping when the result looks good, which is the problem described under the risk of peeking and stopping rules.

Limits

• The gate cannot make a weak pilot strong. A small pilot with a demanding threshold may fail to meet the gate even when the effect is real. Simulating the pilot design under plausible true effects before the pilot, and writing the result next to the gate, shows how often each outcome should be expected.
• Margins are risk decisions. Relaxing a guardrail after seeing a near miss should be a decision by the people who own that risk, recorded as a change to the rule, not an analyst’s adjustment.
• Deviations happen. Data can be lost, or a metric can turn out to be defined badly. Record the deviation and its reason, and report which parts of the analysis followed the plan.
• It limits the analysis, not the story. A result can still be described selectively after the rule has returned its answer. Reporting every pre-specified result guards against that.

The practice comes from clinical trials and from pre-registration in science, where registering a plan separates analyses that test a prediction from analyses suggested by the data. For the quantities a gate typically reads, see confidence intervals and hypothesis testing.

References

• Brian A. Nosek, Charles R. Ebersole, Alexander C. DeHaven, and David T. Mellor, The Preregistration Revolution (PNAS, 2018)