Comparing Two Independent Proportions and Means

Two-sample inference evaluates whether an observed difference between two groups or time points is larger than we would reasonably expect from random sampling variation.

Common applications include:

• Comparing approval ratings from two independent polls
• Comparing treatment and control groups
• Comparing conversion rates between website designs
• Comparing disease rates across populations
• Comparing average outcomes between two groups

The appropriate method depends on whether the outcomes are proportions or quantitative means and whether the samples are independent or paired.

Comparing Two Approval Ratings

Suppose two independent polls report:

Poll	Sample size	Approval count	Approval proportion
Last month	\(n_1=1{,}000\)	\(x_1=550\)	\(\hat p_1=0.55\)
This month	\(n_2=1{,}500\)	\(x_2=870\)	\(\hat p_2=0.58\)

The observed change is\[ \hat p_2-\hat p_1 = 0.58-0.55 = 0.03. \]

On the percentage scale, this is an increase of\[ 3\text{ percentage points}. \]

The statistical question is whether this difference provides convincing evidence that the corresponding population approval proportions changed.

Defining the Parameters

Let\[ p_1 = \text{population approval proportion last month} \]

and\[ p_2 = \text{population approval proportion this month}. \]

For a two-sided test, the hypotheses are\[ \boxed{ H_0:p_1=p_2 } \]

and\[ \boxed{ H_A:p_1\neq p_2 }. \]

Equivalently,\[ H_0:p_2-p_1=0 \]

versus\[ H_A:p_2-p_1\neq0. \]

A two-sided alternative is appropriate when either an increase or a decrease would count as evidence of change.

If the direction was specified before examining the data and only an increase is relevant, a one-sided alternative could be\[ H_A:p_2>p_1. \]

Sampling Distribution of a Difference

Each sample proportion has an approximately normal sampling distribution when the expected success and failure counts are sufficiently large.

For independent samples,\[ \operatorname{Var}(\hat p_2-\hat p_1) = \operatorname{Var}(\hat p_2) + \operatorname{Var}(\hat p_1). \]

The covariance term is zero because of independence.

Thus,\[ SE(\hat p_2-\hat p_1) = \sqrt{ SE(\hat p_1)^2 + SE(\hat p_2)^2 }. \]

Using the population proportions,\[ SE(\hat p_2-\hat p_1) = \sqrt{ \frac{p_1(1-p_1)}{n_1} + \frac{p_2(1-p_2)}{n_2} }. \]

Since \(p_1\) and \(p_2\) are unknown, they must be estimated.

Two Different Standard Errors

The standard error used for a confidence interval is generally not exactly the same as the one used for the null-hypothesis significance test.

Unpooled Standard Error

For estimating the unrestricted difference \(p_2-p_1\), use the separate sample proportions:\[ \boxed{ \widehat{SE}_{\text{unpooled}} = \sqrt{ \frac{\hat p_1(1-\hat p_1)}{n_1} + \frac{\hat p_2(1-\hat p_2)}{n_2} } }. \]

This is typically used for a confidence interval.

Pooled Standard Error

Under the null hypothesis,\[ p_1=p_2=p. \]

Because the proportions are assumed equal under \(H_0\), both samples can be combined to estimate their common value:\[ \boxed{ \hat p_{\text{pool}} = \frac{x_1+x_2}{n_1+n_2} }. \]

The null standard error is then\[ \boxed{ \widehat{SE}_0 = \sqrt{ \hat p_{\text{pool}} (1-\hat p_{\text{pool}}) \left( \frac{1}{n_1} + \frac{1}{n_2} \right) } }. \]

This pooled standard error is used in the conventional two-proportion z-test.

Calculating the Pooled Proportion

The first sample contains\[ x_1=0.55(1{,}000)=550 \]

approvals.

The second contains\[ x_2=0.58(1{,}500)=870 \]

approvals.

The combined number of approvals is\[ 550+870=1{,}420. \]

The combined sample size is\[ 1{,}000+1{,}500=2{,}500. \]

Therefore,\[ \hat p_{\text{pool}} = \frac{1{,}420}{2{,}500} = 0.568. \]

Thus,\[ \boxed{ \hat p_{\text{pool}}=56.8\% }. \]

Calculating the Null Standard Error

The pooled standard error is\[ \widehat{SE}_0 = \sqrt{ 0.568(1-0.568) \left( \frac{1}{1{,}000} + \frac{1}{1{,}500} \right) }. \]

Since\[ 1-0.568=0.432, \]

we have\[ \widehat{SE}_0 = \sqrt{ 0.568(0.432) \left( 0.001+\frac{1}{1{,}500} \right) }. \]

This gives\[ \boxed{ \widehat{SE}_0\approx0.02022 }. \]

On the percentage scale, the standard error is approximately\[ 2.02\text{ percentage points}. \]

Calculating the z-Statistic

The general test statistic is\[ z = \frac{ \text{observed difference} – \text{null difference} }{ \text{null standard error} }. \]

Here,\[ z = \frac{ (0.58-0.55)-0 }{ 0.02022 }. \]

Therefore,\[ z = \frac{0.03}{0.02022} \approx 1.48. \]

Thus,\[ \boxed{z\approx1.48}. \]

The observed increase is approximately 1.48 standard errors above the zero difference specified by the null hypothesis.

Calculating the P-Value

For the two-sided alternative,\[ H_A:p_2-p_1\neq0, \]

the p-value is\[ P(|Z|\geq1.48). \]

By symmetry,\[ \text{p-value} = 2P(Z\geq1.48). \]

The upper-tail probability is approximately\[ P(Z\geq1.48)\approx0.069. \]

Therefore,\[ \boxed{ \text{p-value}\approx0.138 }. \]

Rounded, the p-value is approximately 14%.

At the conventional 5% significance level,\[ 0.138>0.05, \]

so we fail to reject \(H_0\).

Interpreting the Test

The data do not provide sufficiently strong evidence, at the 5% level, that the population approval proportion changed between the two months.

This does not prove that the approval rating remained exactly constant.

The observed difference is\[ 3\text{ percentage points}, \]

but the estimated sampling uncertainty is large enough that a zero difference remains plausible.

A nonsignificant result can mean:

• There is no change.
• There is a small change.
• There is a moderate change that the samples cannot estimate precisely.
• Sampling and survey-design variation obscured the change.

The confidence interval helps show which differences remain compatible with the data.

Confidence Interval for the Difference

For a confidence interval, use the unpooled standard error:\[ \widehat{SE}_{\text{unpooled}} = \sqrt{ \frac{0.55(0.45)}{1{,}000} + \frac{0.58(0.42)}{1{,}500} }. \]

The first component is\[ \frac{0.55(0.45)}{1{,}000} = 0.0002475. \]

The second is\[ \frac{0.58(0.42)}{1{,}500} = 0.0001624. \]

Therefore,\[ \widehat{SE}_{\text{unpooled}} = \sqrt{0.0004099} \approx 0.02025. \]

The 95% confidence interval is\[ (\hat p_2-\hat p_1) \pm 1.96\widehat{SE}_{\text{unpooled}}. \]

Thus,\[ 0.03 \pm 1.96(0.02025). \]

The margin of error is approximately\[ 1.96(0.02025) \approx 0.0397. \]

The interval is\[ 0.03\pm0.0397, \]

or\[ \boxed{ (-0.0097,\ 0.0697) }. \]

Expressed in percentage points:\[ \boxed{ -1.0\text{ to }7.0\text{ percentage points} }. \]

Interpreting the Confidence Interval

The interval contains zero, which agrees with the nonsignificant two-sided test.

The data are compatible with population changes ranging approximately from:

• A decline of 1 percentage point
• To an increase of 7 percentage points

Therefore, the data do not rule out no change. They also do not rule out a potentially meaningful increase.

“Not statistically significant” is not equivalent to “no important change.”

Pooled Test and Unpooled Interval

The two-proportion test uses pooling because the null hypothesis asserts\[ p_1=p_2. \]

The confidence interval does not assume equality; it estimates the unrestricted difference\[ p_2-p_1. \]

Consequently:

Procedure	Standard error
Test of \(H_0:p_1=p_2\)	Pooled
Confidence interval for \(p_2-p_1\)	Unpooled

In this example, the pooled and unpooled standard errors are nearly identical:\[ 0.02022 \]

versus\[ 0.02025. \]

The conceptual distinction remains important even when the numerical difference is small.

Conditions for the Two-Proportion z-Test

The conventional procedure requires:

Independent Samples

The respondents in one poll should be independent of those in the other, or the design must account for dependence.

Independent Observations Within Samples

Responses should be approximately independent within each poll, subject to the actual survey design.

Adequate Success and Failure Counts

Using the pooled estimate for the test, expected counts should be sufficiently large:\[ n_1\hat p_{\text{pool}}, \quad n_1(1-\hat p_{\text{pool}}), \quad n_2\hat p_{\text{pool}}, \quad n_2(1-\hat p_{\text{pool}}). \]

All are much larger than 10 in this example.

Comparable Target Definitions

The polls should use comparable definitions of:

• Likely voters
• Approval
• Geographic coverage
• Survey mode
• Weighting
• Field dates

An observed difference can reflect methodological changes rather than actual opinion change.

What If the Same Voters Are Surveyed Twice?

If the same individuals are surveyed in both months, the samples are paired rather than independent.

The ordinary two-proportion z-test is then inappropriate because it ignores within-person dependence.

For paired binary responses, the relevant information is the number of respondents who changed:

• From disapproval to approval
• From approval to disapproval

McNemar’s test is commonly used for such paired binary data.

The paired structure can improve precision because each respondent serves as their own comparison.

Comparing Two Independent Means

Suppose the response is quantitative rather than binary.

Let:\[ \mu_1,\mu_2 \]

be two population means, with sample estimates\[ \bar X_1,\bar X_2. \]

For independent samples,\[ \operatorname{Var}(\bar X_1-\bar X_2) = \operatorname{Var}(\bar X_1) + \operatorname{Var}(\bar X_2). \]

Therefore,\[ SE(\bar X_1-\bar X_2) = \sqrt{ \frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2} }. \]

Because the population standard deviations are generally unknown, estimate them with \(s_1\) and \(s_2\):\[ \boxed{ \widehat{SE}(\bar X_1-\bar X_2) = \sqrt{ \frac{s_1^2}{n_1} + \frac{s_2^2}{n_2} } }. \]

Welch’s Two-Sample t-Test

The generally preferred method for comparing two independent means is Welch’s t-test.

For\[ H_0:\mu_1-\mu_2=\Delta_0, \]

the test statistic is\[ \boxed{ t = \frac{ (\bar X_1-\bar X_2)-\Delta_0 }{ \sqrt{ s_1^2/n_1+s_2^2/n_2 } } }. \]

The degrees of freedom are approximated using the Welch–Satterthwaite formula:\[ \nu = \frac{ \left( s_1^2/n_1+s_2^2/n_2 \right)^2 }{ \frac{(s_1^2/n_1)^2}{n_1-1} + \frac{(s_2^2/n_2)^2}{n_2-1} }. \]

Statistical software calculates this automatically.

Welch’s test does not require equal population variances and performs well even when the variances happen to be equal. It is therefore a sensible default.

The Pooled Two-Sample t-Test

The pooled t-test assumes\[ \sigma_1^2=\sigma_2^2=\sigma^2. \]

The common variance is estimated by\[ s_p^2 = \frac{ (n_1-1)s_1^2 + (n_2-1)s_2^2 }{ n_1+n_2-2 }. \]

The pooled standard error is\[ SE_{\text{pooled}} = s_p \sqrt{ \frac{1}{n_1} + \frac{1}{n_2} }. \]

The degrees of freedom are\[ n_1+n_2-2. \]

Pooling can offer a modest efficiency gain if the equal-variance assumption is correct. If it is wrong—especially with unequal sample sizes—the results can be misleading.

For this reason, Welch’s test is usually preferred unless equal variances are scientifically well justified.

Is a z-Test Appropriate for Two Means?

If \(\sigma_1\) and \(\sigma_2\) are truly known, a two-sample z-test can be used.

In most real applications, they are unknown and are estimated from the sample. A t-based procedure is then appropriate. This remains conceptually true even with large samples, although the \(t\)-distribution becomes nearly indistinguishable from the standard normal distribution.

Thus, “use a z-test when samples are large” is a useful approximation, but Welch’s t-test remains a sound default when population variances are unknown.

Independent Treatment and Control Groups

When different participants are randomly assigned to treatment and control groups, their outcomes are normally analyzed as two independent groups, assuming one participant’s outcome does not affect another’s.

Random assignment does not make the two outcome samples paired. Pairing arises when:

• The same subject is measured twice
• Participants are explicitly matched
• Outcomes occur in natural pairs
• Cluster or repeated-measure structures connect observations

A randomized experiment may still require specialized analysis if participants are assigned by cluster, outcomes interfere across participants, or blocking and matching were used.

Independence Cannot Be Decided from Labels Alone

Two samples are not independent merely because they have different names, and they are not paired merely because they were measured at two times.

The study design determines the relationship.

Design	Appropriate structure
Different voters sampled each month	Independent samples
Same voters surveyed both months	Paired binary data
Different patients randomized individually	Independent groups
Same patients measured before and after treatment	Paired quantitative data
Schools randomized as groups	Clustered data
Participants matched in pairs before randomization	Matched-pair analysis

Survey Design and Total Error

The formulas above describe simple random-sampling variation. Real polls may involve:

• Survey weights
• Stratification
• Clustering
• Nonresponse adjustments
• Likely-voter models
• Changing survey modes
• Different questionnaire wording

A design-adjusted standard error may differ from the simple formula.

Moreover, a confidence interval does not automatically include systematic errors such as nonresponse or coverage bias. Two polls can differ because of methodology even if the underlying public opinion has not changed.

Statistical Significance Versus Meaningful Change

The observed increase is\[ 3\text{ percentage points}. \]

The p-value is approximately\[ 0.14. \]

This indicates that the data do not provide strong evidence against equal population proportions under the simple independent-sampling model.

It does not determine whether a 3-point change would be politically meaningful. Practical importance and statistical certainty are different questions.

The confidence interval\[ (-1.0,\ 7.0) \]

percentage points communicates both:

• The estimated direction and magnitude
• The remaining uncertainty

Key Takeaway

For two independent sample proportions, the observed difference is\[ \hat p_2-\hat p_1. \]

The conventional test of\[ H_0:p_1=p_2 \]

uses the pooled estimate\[ \hat p_{\text{pool}} = \frac{x_1+x_2}{n_1+n_2} \]

and test statistic\[ z = \frac{ \hat p_2-\hat p_1 }{ \sqrt{ \hat p_{\text{pool}} (1-\hat p_{\text{pool}}) \left( 1/n_1+1/n_2 \right) } }. \]

For approval ratings of 55% from 1,000 voters and 58% from 1,500 voters,\[ \boxed{z\approx1.48} \]

and the two-sided p-value is approximately\[ \boxed{0.14}. \]

The 95% confidence interval for the change is approximately\[ \boxed{ -1.0\text{ to }7.0\text{ percentage points} }. \]

The interval contains zero, so the data do not provide strong evidence of a change. This analysis assumes independent samples. If the same respondents are observed twice, a paired method such as McNemar’s test is required. For two independent quantitative means with unknown variances, Welch’s t-test is generally the preferred procedure.