Sample size

Sample size, written \(n\) or \(N\), counts observations or sampled units, which may be dependent. For independent observations with a common variance, familiar formulas relate precision to that count — the standard error shrinks like \(1/\sqrt{n}\), the margin of error with it, and power rises with it.

Two things about it cause nearly all the trouble in practice: the word independent, and the fact that there is no correct value without assumptions.

n is not the number of rows

The simple formulas below assume independent draws. Distinguish the row count, the number of independent sampling or assignment units, and effective information after accounting for dependence. Treating dependent rows as independent can understate or overstate uncertainty, depending on the covariance and estimator.

You have	The question is about	\(n\) is
10,000 events from 2,000 users	Users	2,000 user units if users are independent; event-level analysis must account for within-user dependence and weighting
90 days of a daily metric	The daily level	90 observations; effective information depends on the covariance structure and can be lower or higher than under independence. See temporal autocorrelation
500 responses from 120 households	Households or people	Depends on which, and the design has to be accounted for

The rule is to name the unit of randomization before counting. In an experiment where assignment happens per user, the unit is the user even if the metric is per event — an event-level analysis is valid only if it accounts for that assignment and dependence structure, for example with an appropriate cluster-aware estimator. User-level aggregation is another option, but user and event weighting can target different estimands; neither aggregation nor a large table automatically establishes independence. See IID for the assumption being relied on, and where a closed form is awkward, bootstrap at the level of the correct unit.

There is no right size without assumptions

The NIST handbook answers the most common request bluntly: asked how many measurements to include, “unfortunately, there is no correct answer without additional information (or assumptions).” What you need to supply is \(\alpha\), \(\beta\), and the population standard deviation.

For a one-sample mean compared with a fixed reference, with independent normal observations and known population standard deviation \(\sigma\), the usual two-sided power approximation is (with target power \(1-\beta\) for a shift \(\delta\)):

\[N = (z_{1-\alpha/2} + z_{1-\beta})^2 \left(\frac{\sigma}{\delta}\right)^2\]

Here \(N\) counts independent observations; for paired differences it counts independent pairs and \(\sigma\) is the standard deviation of the differences. For two independent groups, start from \(SE(\bar X_A-\bar X_B)=\sqrt{\sigma_A^2/n_A+\sigma_B^2/n_B}\). With equal variances and equal allocation, the corresponding normal approximation needs twice the one-sample \(N\) per group, four times that \(N\) in total. For example, \(\alpha=.05\), power .80, \(\sigma=1\), and \(\delta=.2\) give about 196.22 for one sample, 392.44 per group, or 784.89 total before rounding up. An actual design must account for variance estimation and allocation; a conversion-rate comparison also needs the baseline rate and an absolute or relative lift definition.

To merely estimate something to within \(\delta\) at 95%: \(N \ge (1.96/\delta)^2\sigma^2\). The first is a testing question and the second a precision question; deciding which you are asking comes before the arithmetic. Note also that the handbook states the convention of expressing \(\delta\) “in units of the standard deviation” — which makes the input an effect size and the whole calculation a matter of \(\sigma/\delta\).

Both are quadratic in \(1/\delta\), which is the single most useful fact to internalize about sample size.

• Detecting an effect half as large needs about four times the data.
• Halving the margin of error needs about four times the data.
• Quadrupling the sample halves the standard error — not quarters it.

The dependence on \(\sigma\) is the input teams skip, and the handbook flags the constraint: “a restriction is that the standard deviation must be known,” with the fallback being “the best estimate available from a previous experiment.” Because \(N\) scales with \(\sigma^2\), an optimistic guess at variability produces a study that is quietly too small. See power analysis for the calculation and sample standard deviation for the estimate it needs.

When the size is fixed

Often \(n\) is not a choice — you get the traffic you get, or the patients, or the four quarters of history. Then the equation is more useful read backwards: given this sample, what is the smallest effect I could reliably detect? That is a minimum detectable lift, and computing it before running helps assess the chance of detecting an important effect at the chosen target power and the value of the expected precision.

It also disciplines the alternative shortcuts. Loosening \(\alpha\) buys power at the cost of false positives — “the risk \(\beta\) increases as the risk \(\alpha\) decreases,” and the reverse — while stopping as soon as a result looks good is not a shortcut but a different procedure with a different error rate. See risk of peeking and stopping rules.

What more data does not fix

A larger sample reduces sampling variability and nothing else. It does not remove bias: a sample that systematically misses some population converges on the wrong value more precisely, and a metric defined incorrectly is computed more precisely too. It does not repair a confounded comparison — see causal effect.

And it has a failure mode of its own: with a large enough \(n\), differences too small to act on become statistically significant, which is why magnitude has to be reported beside the verdict. See trivial effects and p-value.

Further reading

• NIST/SEMATECH e-Handbook of Statistical Methods, §7.2.2.2 “Sample sizes required”
• §7.1.3 “What are statistical tests?”

For two-group sizing, see statsmodels, TTestIndPower.power: group sample sizes, allocation ratio, and standardized effect.