Rule of Three

The rule of three says that if you observe an event in zero of \(n\) independent trials, an approximate 95% upper confidence bound on the event's probability is \(3/n\). It answers the question "we saw no failures; how bad could the true failure rate still be?" Hanley and Lippman-Hand described it in a 1983 JAMA article on interpreting zero numerators.

Derivation

Suppose each trial fails independently with the same unknown probability \(p\). The probability of seeing no failures in \(n\) trials is

\[ P(\text{0 failures}) = (1-p)^n. \]

A value of \(p\) is plausible given the data if this probability is not too small. Call \(p\) implausible when seeing zero failures would have had probability at most 5%. Solving \((1-p)^n = 0.05\) gives the exact upper bound

\[ p_{\text{upper}} = 1 – 0.05^{1/n}. \]

For large \(n\) and small \(p\), \((1-p)^n \approx e^{-np}\). Setting \(e^{-np} = 0.05\) gives \(np = -\ln 0.05 = 2.9957\), which rounds to 3, so \(p_{\text{upper}} \approx 3/n\). This is a one-sided bound: it says the rate is likely between 0 and \(3/n\), with 95% confidence in the usual sense.

How close the approximation is

Computed with the exact formula:

n	exact 95% bound	3/n
10	0.25887	0.30000
30	0.09503	0.10000
100	0.02951	0.03000
300	0.00994	0.01000
1000	0.00299	0.00300
10000	0.00030	0.00030

The rule is slightly conservative and agrees closely from about \(n=30\) on, which is the range often quoted for it. For small \(n\) use the exact formula.

Using it

To show with 95% confidence that a rare failure occurs at a rate below a target \(p_0\), you need about \(3/p_0\) failure-free trials. For \(p_0 = 1\%\), that is 300 trials (the exact formula gives 299). Going ten times lower costs ten times as many trials: showing a rate under 0.1% needs about 3000. Zero failures in 100 independent trials give an exact one-sided 95% upper bound of 2.951%, just below 3%, whereas the rule-of-three approximation gives 3%. This is weak evidence for substantially lower rates, and a team that sees none in 20 should not conclude that the failure is negligible, since \(3/20\) is 15%.

This gives a rough planning rule for release gates. For a hypothetical agent action that must stay below a 1% severe-failure rate, a stage of canary traffic or an evaluation set with 300 independent runs and no failures is the minimum evidence. One failure in 300 changes the calculation and needs an interval method for nonzero counts, for example the exact binomial (Clopper-Pearson) interval.

What the 95% means

A simulation with a true rate \(p = 0.0105\) and \(n = 300\), repeated 20,000 times, saw zero failures in about 4.0% of experiments (theory: \((1-0.0105)^{300} = 4.2\%\)). In those runs the bound \(3/n = 0.0100\) was below the true rate, which is the 5%-at-most failure the procedure accepts. The confidence level describes the procedure across many experiments, not a probability that a given \(p\) is in a given interval.

Conditions

• Trials must be independent and share one failure probability. Sessions from the same user, a shared dependency, or traffic that changes over time break this and make the bound too optimistic. Count independent units, for example one session per user, or use a method for clustered data.
• It addresses zero observed events only. With \(k>0\) failures, use a binomial interval.
• It bounds the rate in the sampled population. Failures that appear only under conditions the trials never exercised, such as a rare tool path, are not covered.
• A bound is a statement about the rate, not about severity. A failure that is irreversible may need a much smaller target than the 1% in the example.

References

• Hanley and Lippman-Hand, If Nothing Goes Wrong, Is Everything All Right? Interpreting Zero Numerators, JAMA 249(13), 1983