Design Effect

The design effect is the factor by which clustering inflates the variance of an estimate, compared with a sample of the same size in which every observation is independent. It matters because many datasets contain groups of related items: messages in one thread, questions about one document, variants of one template. Items in a group share difficulty or properties, so each additional item from the same group carries less new information than an independent one, and an estimate based on \(n\) rows is less precise than the row count suggests.

The formula

Suppose the \(n\) items fall into clusters of equal size \(m\). Let \(\rho\) be the intraclass correlation: the correlation between two items in the same cluster (0 if items in a cluster are no more alike than items from different clusters, approaching 1 if they are interchangeable). The variance of the sample mean is multiplied by

\[ \text{DEFF}=1+(m-1)\rho . \]

Dividing the sample size by the design effect gives the effective sample size,

\[ n_\text{eff}=\frac{n}{1+(m-1)\rho}. \]

\(n_\text{eff}\) is the number of independent observations that would give the same precision. Confidence intervals widen by \(\sqrt{\text{DEFF}}\), since a standard error is a square root of a variance. The formula follows Kish's treatment of cluster sampling and is exact for the mean under equal cluster sizes and a common \(\rho\); unequal sizes and varying correlations make it an approximation.

Reading the formula

• With \(\rho=0\) the factor is 1 and clustering costs nothing.
• With \(\rho=1\) each cluster is one observation repeated \(m\) times, and \(n_\text{eff}=n/m\), the number of clusters.
• In between, the loss grows with both the cluster size and the correlation. A small \(\rho\) matters when clusters are large, because the term is \((m-1)\rho\).

Worked example

Take 300 evaluation items in 30 clusters of 10 with \(\rho=0.3\) (a hypothetical set). Then \(\text{DEFF}=1+9\times0.3=3.7\) and \(n_\text{eff}=300/3.7\approx81\), so the 300 rows behave like about 81 independent ones, and intervals are about \(\sqrt{3.7}=1.92\) times as wide as the independent formula gives.

A simulation checks the variance claim: draw each item as a cluster-level effect (variance \(\rho\)) plus individual noise (variance \(1-\rho\)), so each item has variance 1 and within-cluster correlation 0.3. Over 4,000 repeated samples, the variance of the mean was 0.01227 for 30 clusters of 10, against 0.00326 for 300 independent items, a ratio of 3.77, close to the predicted 3.7. How the effective size falls with \(\rho\) for the same 300 items in clusters of 10:

\(\rho\)	0	0.05	0.1	0.3	0.6
\(n_\text{eff}\)	300	207	158	81	47

Consequences and remedies

• A test set of many items from few clusters gives intervals that look narrow when computed by the independent formula and are too optimistic. The fix is to compute intervals that respect the clusters: a cluster bootstrap that resamples whole groups, or the design-effect adjustment above.
• When planning an evaluation, the number of clusters constrains precision more than the number of rows. Adding more items from the same threads helps little; adding threads helps.
• To estimate \(\rho\), compare variation between clusters to variation within them (one-way analysis of variance gives the intraclass correlation). A guessed \(\rho\) is acceptable for planning but should be stated.
• The equal-size formula describes the variance of a mean over items. Scoring the same items with two models allows shared item difficulty to cancel in their paired differences, which often makes a paired comparison more precise. Differences can still be correlated within a document or thread, so apply the cluster bootstrap or a design-effect calculation to those differences rather than treating pairing as a cure for clustering.

References

• Kish, Survey Sampling (1965), the source of the design effect.