Compromise Power Analysis

Compromise power analysis chooses a critical value by specifying the ratio of two error probabilities at a fixed sample size and assumed effect. The usual calculation fixes the significance level and the desired power and solves for the sample size. When the sample is fixed — a week of traffic, a fixed cohort, a dataset that already exists — first check whether it supports the desired significance level and power for the assumed effect. If it does not, compromise analysis makes the trade-off explicit by solving for both error probabilities through their ratio.

What is fixed and what is solved for

The analysis takes the sample size and the effect size as given, and takes one more input: the ratio \(\beta/\alpha\) you are willing to accept between the two error rates. G*Power’s own description is precise about what comes out: “The statistical decision criterion (‘critical value’) and the associated \(\alpha\) and \(\beta\) values are computed as a function of the desired error probability ratio \(\beta/\alpha\), the sample size, and the population effect size.”

So the output is a cut-off, and the two error rates are what that cut-off implies. Moving the critical value down raises the false-positive rate and lowers the false-negative rate; moving it up does the reverse. With the sample fixed, that is the only lever, and the ratio is the statement of how the analyst wants the loss divided.

The middle three rows all hold something at a convention. Compromise analysis is the one that refuses to, and it is the honest choice when the convention has stopped being affordable.

A worked example

A one-sided two-sample comparison, a standardized effect of 0.5, and twenty units per group.

import numpy as np
from scipy import optimize, stats

def compromise(effect, n_per_group, ratio=1.0):
    """Critical value and the two error rates for a one-sided two-sample z test,
    given the sample size and the desired ratio beta/alpha."""
    noncentrality = effect * np.sqrt(n_per_group / 2)
    def imbalance(c):
        alpha = 1 - stats.norm.cdf(c)
        beta = stats.norm.cdf(c - noncentrality)
        return beta - ratio * alpha
    c = optimize.brentq(imbalance, -10, 10)
    return c, 1 - stats.norm.cdf(c), stats.norm.cdf(c - noncentrality)

for ratio in (1.0, 4.0):
    c, alpha, beta = compromise(effect=0.5, n_per_group=20, ratio=ratio)
    print(f"{ratio:.1f} {c:.4f} {alpha:.4f} {beta:.4f} {1 - beta:.4f}")
# 1.0 0.7906 0.2146 0.2146 0.7854
# 4.0 1.2980 0.0971 0.3885 0.6115

conventional_alpha = 0.05
# Holding alpha at the convention instead, with the same 20 per group.
power = 1 - stats.norm.cdf(stats.norm.ppf(1 - conventional_alpha) - 0.5 * np.sqrt(20 / 2))
print(f"{conventional_alpha:.4f} {1 - power:.4f} {power:.4f}")
# 0.0500 0.5254 0.4746

gpower_critical_z = 1.385675
# The critical value G*Power reports for its own compromise example, checked.
print(f"{1 - stats.norm.cdf(gpower_critical_z):.6f}")
# 0.082923

The third row is what actually happens when a team with forty units runs the standard test: a 5 percent false-positive rate bought at the price of missing a real effect of this size more than half the time. The study is not so much a test as a coin flip with good manners. The first row keeps the two mistakes equally likely at about 21 percent each, which is a worse significance level and a far better chance of noticing the effect.

Which row is right depends entirely on what the two mistakes cost. A screening study that feeds into confirmation can tolerate false positives and should not tolerate missing candidates. A result that will be published as a finding, or shipped without further checking, cannot.

What it does not do

Compromise analysis adds no information. It trades one error probability against the other at a fixed sample size and effect; what is fixed is the trade-off curve, not the sum \(\alpha + \beta\). And the ratio \(\beta/\alpha\) is not a cost ratio: minimizing expected cost requires the prior odds as well as the losses, and in this example a genuine 4:1 cost ratio at equal priors puts the critical value at 1.6673 rather than the 1.2980 the compromise returns. Choosing a ratio is a statement about which error probability you will tolerate, not a way to make a small study behave like a large one, and presenting the result as though the study were now adequately powered misreads it entirely.

Two further limits are worth stating. The calculation depends on an assumed effect size, and with a small sample the answer is sensitive to it — the numbers above are for an effect of 0.5, and halving that assumption changes every row. And journals and review processes mostly expect \(\alpha = 0.05\), so a study reporting \(\alpha = 0.21\) needs its reasoning in the text rather than in a footnote. That is a communication problem rather than a statistical one, but it is the reason the method is rarer than the situations calling for it.

The alternative, when the result matters and the sample cannot grow, is not a different calculation. It is a different design: reducing variance, using a paired or within-subject comparison, choosing a more sensitive outcome, or accepting that the question cannot be answered with the data available. Statistical power covers the general relationship, minimum detectable lift the effect a fixed design can see, and sample size the arithmetic underneath both.

References

• Franz Faul, Edgar Erdfelder, Axel Buchner and Albert-Georg Lang, “Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses”, Behavior Research Methods 41(4), 1149–1160, 2009