Noninferiority Margin

A noninferiority margin \(m\) is the largest loss in a metric that you are willing to accept in exchange for the other benefits of a new option, fixed before the data are examined. A noninferiority test is a one-sided hypothesis test that asks whether the data show that the new option is worse than the current one by no more than \(m\). It is the right question for a cheaper, faster, or simpler candidate: nobody needs it to be better, but it must not be much worse.

The hypotheses and the interval rule

Let \(\Delta\) be the true difference, candidate minus baseline, in a metric where higher is better. The null hypothesis is that the candidate is inferior by at least the margin, \(H_0: \Delta \le -m\), and the alternative is \(H_1: \Delta > -m\). A rejection of \(H_0\) is the evidence for noninferiority. Compare this with an ordinary test of equality. There, a small sample produces “no significant difference”, which reads as reassurance but only reflects too little data. Here the burden is reversed, and a small or noisy sample leaves the interval wide, so the test cannot reject and the candidate does not pass.

The test has a direct interval form. A one-sided test at level \(\alpha\) rejects \(H_0\) exactly when the one-sided \(1 – \alpha\) lower confidence bound lies above \(-m\). The lower end of a two-sided 95 percent interval is a one-sided 97.5 percent lower bound, so checking that \(\text{lo} \ge -m\) on a two-sided 95 percent interval corresponds to a one-sided test with \(\alpha = 0.025\). The upper end is irrelevant to the decision. The confidence interval can come from a normal approximation, as below, or from a paired bootstrap.

The calculation below applies a 3-point margin to pass-rate differences measured on 3,000 paired examples, with a standard deviation of per-example differences of 0.47. The second part gives the number of examples needed for 90 percent power at \(\alpha = 0.025\), using \(n = \left((z_{1-\alpha} + z_{1-\beta})\, s / (m + \Delta)\right)^2\), where \(\Delta\) is the assumed true difference. The last line is a hypothetical sequence of five 2-point losses that illustrates cumulative drift; it does not apply a test to those releases.

from math import sqrt, ceil
from statistics import NormalDist
z = NormalDist().inv_cdf
m, s, n = 0.03, 0.47, 3000          # margin 3 points; sd of per-example paired differences; examples
alpha, power = 0.025, 0.90

def lower_bound(d, s, n, alpha=0.025):
    return d - z(1 - alpha) * s / sqrt(n)

for d in (0.01, -0.005, -0.02, -0.04):
    lo = lower_bound(d, s, n)
    print(f"observed diff {d*100:+.1f} pts  lower bound {lo*100:+.2f} pts  non-inferior at margin 3 pts: {lo >= -m}")

n_equal = ((z(1 - alpha) + z(power)) * s / m) ** 2
print(f"examples for {power:.0%} power when the true difference is 0: {ceil(n_equal)}")
n_worse = ((z(1 - alpha) + z(power)) * s / (m - 0.01)) ** 2
print(f"examples for {power:.0%} power when the candidate is truly 1 point worse: {ceil(n_worse)}")

# hypothetical drift: five 2-point losses; no noninferiority tests are applied here
score = 0.80
for v in range(1, 6): score -= 0.02; print(f"v{v}: {score:.2f}", end="  ")
print()

observed diff +1.0 pts  lower bound -0.68 pts  non-inferior at margin 3 pts: True
observed diff -0.5 pts  lower bound -2.18 pts  non-inferior at margin 3 pts: True
observed diff -2.0 pts  lower bound -3.68 pts  non-inferior at margin 3 pts: False
observed diff -4.0 pts  lower bound -5.68 pts  non-inferior at margin 3 pts: False
examples for 90% power when the true difference is 0: 2579
examples for 90% power when the candidate is truly 1 point worse: 5803
v1: 0.78  v2: 0.76  v3: 0.74  v4: 0.72  v5: 0.70

A candidate observed 0.5 points below the baseline passes because its lower bound is at -2.18 points. A candidate observed 2 points below fails, even though 2 is smaller than the margin of 3: with this sample size the interval reaches -3.68 points, so a loss larger than the margin remains plausible. The margin does not move the observed difference. It sets the line the lower bound has to clear. Showing noninferiority for a truly equal candidate needs about 2,600 examples here, and for a candidate that is truly 1 point worse the requirement more than doubles. A tight margin, a noisy metric, and a small evaluation set together make the test unusable.

Choosing the margin

The margin is a decision about tolerable loss, so it comes from the people who own the consequences, in the unit of the metric and in the right direction for a lower-is-better metric. It is fixed before the comparison and not adjusted until the candidate passes. It should not be set to the measurement noise or to whatever the evaluation set can verify. If the margin the product needs cannot be verified with the available data, the answer is a larger evaluation set, not a wider margin. Regulatory guidance for clinical trials, such as the FDA’s guidance on noninferiority trials, covers how to choose the margin and how to test the hypothesis as separate questions.

Each noninferiority comparison is relative to its baseline. If every release is compared only with the previous one, small losses can accumulate even when each comparison passes. The hypothetical five-release sequence above illustrates the accumulation alone, ending 10 points below the initial 0.80 score; the earlier calculation shows that a 2-point loss does not pass with 3,000 examples and the stated variability. Comparing against a fixed reference version as well limits the drift. When a large difference in either direction is unacceptable, use an equivalence test with margins on both sides. Noninferiority instead permits a large improvement and tests only against an unacceptable loss.

References

• FDA, Non-Inferiority Clinical Trials guidance (November 2016)