T-Test

A t-test compares a mean, or a difference between means, with its own estimated standard error, and refers the ratio to Student’s t distribution. The estimation is the point. When the population standard deviation is unknown and the sample standard deviation \(s\) stands in for it, the ratio varies more from sample to sample than a normal variable would, and the t distribution with the right number of degrees of freedom accounts for that extra variability. For one sample of \(N\) values with mean \(\bar{Y}\), tested against a hypothesized mean \(\mu_0\), the NIST handbook gives \(t = (\bar{Y} - \mu_0)/(s/\sqrt{N})\) with \(N - 1\) degrees of freedom, and describes estimating the standard deviation from the data as the more typical case. With a known standard deviation the same ratio is a z-test; with a large sample the t distribution is close to the normal and the two usually give the same answer, though the critical values still differ slightly. Both are exact only for normally distributed data. The t-test is therefore not a small-sample method but the usual one whenever \(\sigma\) is estimated.

One-sample, paired, and two-sample

The paired design is worth separating from the two-sample one because it answers a different question with a different amount of noise. When each unit is measured before and after, or two treatments are applied to the same unit, the handbook notes that the difference within each pair is usually what gets analyzed, so the variation between units drops out and only the variation in the changes remains. Treating the same data as two independent samples throws that away and usually loses power; treating two unrelated groups as pairs invents a correspondence that is not there.

For two independent groups the choice is between the pooled statistic and Welch’s. The pooled version assumes the two groups share one standard deviation, an assumption the handbook says should be checked before it is relied on; the Welch version drops it, at the cost of a more complicated formula for the degrees of freedom, which the handbook describes as “robust for unequal sample sizes and/or unequal variances” and which computers make painless. The SciPy function ttest_ind runs the pooled test by default, “a standard independent 2 sample test that assumes equal population variances”, and needs equal_var=False for Welch’s. My recommendation is to use Welch’s version unless there is a specific reason to pool: it gives up little when the variances happen to be equal and keeps the error rate honest when they are not, and it removes a preliminary test whose outcome the analysis would otherwise depend on.

Two worked examples from the handbook

One sample. Particle counts on ten silicon wafers were 50, 48, 44, 56, 61, 52, 53, 55, 67, and 51, with mean 53.7 and standard deviation 6.567, against a long-run process average of 50. The question is whether the process has moved in either direction, so the test is two-sided: \(t = (53.7 - 50)/(6.567/\sqrt{10}) = 1.782\) with 9 degrees of freedom. The two-sided 5 percent critical value is 2.262, taken from the 0.975 column of the t table because the two tails share the 5 percent, and 1.782 is inside it, so the null hypothesis is not rejected. The two-sided p-value is about 0.11, and the 95 percent confidence interval for the mean runs from 49.0 to 58.4, which contains 50.

Two samples. A new assembly procedure was timed on 9 devices against the old one on 11. The old process averaged 36.09 minutes with standard deviation 4.91; the new one 32.22 with standard deviation 2.54. The question is one-sided, whether the new process is faster, and the standard deviations differ by a factor of two, so the Welch statistic is the right one: \(t = 2.269\) with about 15.5 degrees of freedom. The handbook rounds to 16 and reads the one-sided 5 percent critical value, 1.746; the statistic exceeds it, so the new process is judged faster. The one-sided p-value is about 0.019. In code:

from scipy import stats
old = [32, 37, 35, 28, 41, 44, 35, 31, 34, 38, 42]
new = [36, 31, 30, 31, 34, 36, 29, 32, 31]
stats.ttest_ind(old, new, equal_var=False, alternative='greater')
# TtestResult(statistic=2.269..., pvalue=0.0189..., df=15.53...)

The result object carries a confidence_interval() method, and the interval is the same computation as the test seen from the other side, provided direction and level match: the SciPy documentation for the one-sample version states that “The bounds of the 95% confidence interval are the minimum and maximum values of the parameter popmean for which the p-value of the test would be 0.05.” For the one-sided test above the matching object is a one-sided bound, and SciPy returns a lower bound with an infinite upper limit when alternative='greater' is set; a two-sided 95 percent interval belongs with a two-sided test. Reporting the interval alongside the p-value adds no new assumptions and says how large the difference plausibly is, which is the question the test alone leaves open; effect size and practical significance take that further.

Assumptions, and what to do when they fail

The handbook lists the assumptions behind the tests in its process-comparison chapter: the data come from a single process with a single distribution, that distribution is normal, and the observations are uncorrelated over time. On normality it is reassuring within limits: the tests “work fairly well as long as the data are bell-shaped and the tails are not heavy”. A histogram or a normal probability plot is the first check, but with ten observations, as in the wafer example, a plot cannot show whether the tails are heavy, so the judgment also rests on what is known about the measurement process and on whether the conclusion survives a more robust analysis; for a paired test it is the differences, not the two raw samples, whose distribution matters, and a formal normality test is not a substitute for any of this.

Heavy tails and outliers are the case to worry about, and the two standard responses change the question in different ways. SciPy’s trim argument runs a trimmed, or Yuen’s, t-test, which its documentation recommends “if the underlying distribution is long-tailed or contaminated with outliers”; it is a test about the trimmed mean, so it is the right choice only when the tail values are noise rather than part of the quantity the decision depends on.

A permutation p-value, obtained by passing a PermutationMethod, reassigns the observations to the two groups at random many times; it is exact when the null hypothesis makes the groups exchangeable, which for independent samples SciPy states as the data being randomly sampled from the same distribution, so it tests equal distributions rather than equal means alone, and paired data must be permuted within pairs. Welch’s correction, for its part, handles unequal variances and nothing else; it does not repair dependence or a very small, skewed sample.

The assumption that fails most visibly in analytics is not normality but independence. Several sessions from the same user, or several readings from the same day, are not independent observations; when they are positively correlated, a t-test that counts them as independent reports a standard error smaller than the data support, and negative correlation would have the opposite effect. The IID entry covers how to recognize that and what to do; the options include analyzing at the level of the unit that was actually sampled, cluster-robust standard errors, and resampling that keeps the groups intact. How the t-test sits among the other statistical tests, and the logic of rejecting or not rejecting, are covered under hypothesis testing.

References

• NIST/SEMATECH e-Handbook of Statistical Methods, section 7.2.2 Are the data consistent with the assumed process mean?
• NIST/SEMATECH e-Handbook of Statistical Methods, section 7.3.1 Do two processes have the same mean?
• NIST/SEMATECH e-Handbook of Statistical Methods, section 1.3.5.3 Two-Sample t-Test for Equal Means
• NIST/SEMATECH e-Handbook of Statistical Methods, section 7.1.2 What assumptions are typically made?
• SciPy documentation, scipy.stats.ttest_ind
• SciPy documentation, scipy.stats.ttest_1samp
• SciPy documentation, scipy.stats.ttest_rel
• SciPy documentation, scipy.stats.permutation_test