Stratified Sampling

Stratified sampling divides a population into non-overlapping groups called strata, draws a separate random sample inside each stratum, and combines the per-stratum results using each stratum's known share of the population. It is the standard fix when a plain random sample would contain too few cases of a group you care about.

The problem it solves

Suppose a hypothetical request log has three kinds of requests: routine questions (80% of traffic), refund questions (17%), and attempted prompt injections (3%). A simple random sample of 100 requests contains about 3 injections on average. In 20,000 simulated draws of 100 from that population, the median was 3 injection cases and the minimum was 0. The pass rate on the stratum that matters most would be estimated from almost nothing.

Stratified sampling fixes the count per stratum in advance, for example 40 routine, 30 refund and 30 injection, and draws randomly within each.

The estimator

Let \(W_h\) be the share of the population in stratum \(h\), \(n_h\) the number sampled there, and \(\hat p_h\) the observed rate (here, pass rate) in that sample. The overall estimate and its variance are

\[ \hat p = \sum_h W_h\, \hat p_h, \qquad \widehat{\mathrm{Var}}(\hat p) \approx \sum_h W_h^2 \,\frac{\hat p_h(1-\hat p_h)}{n_h}. \]

In words: estimate each stratum separately, then average the stratum rates weighted by how common the stratum really is, not by how many cases you happened to sample from it.

Worked example (hypothetical)

Sampled with 40, 30 and 30 cases, the passes were 38 of 40 (0.95), 24 of 30 (0.80) and 17 of 30 (0.567). The weighted estimate is

\[ 0.80(0.95) + 0.17(0.80) + 0.03(0.567) = 0.913. \]

The unweighted pass rate of the 100 sampled cases is 79/100 = 0.79. That figure is wrong as an estimate of traffic, because injections make up 30% of the sample but 3% of traffic. Oversampling a stratum is fine only if the estimate is reweighted. A later report that quietly averages the raw cases reintroduces the bias.

What it buys, and what it does not

There are two separate benefits.

1. Precision on small strata. With 30 injection cases the standard error of that stratum's rate, at a true rate near 0.6, is \(\sqrt{0.6\cdot0.4/30}\approx 0.089\). With the roughly 3 cases a random sample gives, the same calculation gives about 0.28.
2. Possible variance reduction of the overall estimate. This happens when strata differ in their rates, because the between-stratum variation no longer contributes noise. In this example the gain is small. With the true rates of 0.97, 0.85 and 0.60, the standard deviation of the overall estimate is about 0.024 for a simple random sample of 100, 0.023 for proportional allocation, and 0.024 for the 40/30/30 allocation. The 40/30/30 design did not improve the headline number; it improved the injection estimate. Choose allocation according to which number you need.

Conditions and pitfalls

• The weights \(W_h\) must be known or estimated well. If they come from a stale log, the weighted estimate inherits the error.
• Strata should be defined by something fixed before sampling and related to the outcome, such as request type. Strata chosen by looking at the outcomes defeat the purpose.
• Within each stratum, selection must still be random. Stratifying and then hand-picking examples gives no valid variance formula.
• If \(n_h\) is tiny, the normal-approximation variance above is unreliable; use an exact or Wilson interval for that stratum.
• Stratified sampling is not quota sampling in the loose sense of "include some of each". The difference is random selection within the stratum plus weighted combination.

Related ideas

Cluster sampling samples whole groups and usually increases variance; stratification samples from every group. Importance sampling reweights draws from a deliberately skewed distribution and is the same reweighting idea applied to a continuous setting.