Statistical Tests

A statistical test is a procedure for deciding, from a sample, whether the data give enough evidence against a stated hypothesis about the process that produced them to reject it. The NIST handbook describes the ingredients in a common format: a null hypothesis, which is the claim being doubted; an alternative hypothesis, which is what would be concluded instead; a test statistic computed from the data; a significance level that sets how much false rejection is tolerated; and a critical region, the values of the statistic that lead to rejection. It also states the asymmetry that every test carries: “To reject a hypothesis is to conclude that it is false. However, to accept a hypothesis does not mean that it is true, only that we do not have evidence to believe otherwise.” The logic of that procedure, with its two kinds of error, is the subject of hypothesis testing. This entry is a map of the tests themselves, organized by the question each answers.

Families, by the question asked

The rows follow the handbook’s own grouping of its quantitative techniques, location, scale, randomness, distributional measures, and outliers, with the proportion and correlation tests from its chapter on process comparisons, and the handbook says of its own list that “This list of quantitative techniques is by no means meant to be exhaustive.” Two families sit across the table rather than in one row.

Rank-based tests are often reached for when distributions are far from normal, but they change the question. The Mann–Whitney U test’s null hypothesis, as SciPy states it, is that the two samples come from the same distribution, and it is a test of a difference in location only under the added assumption that the two distributions differ by a shift; the Wilcoxon signed-rank test is the paired counterpart and assumes the paired differences are symmetric about zero. Neither is a test of equal means without those assumptions.

Permutation tests build the null distribution by reassigning observations between groups many times, which is valid when the null hypothesis makes those reassignments exchangeable: for independent samples SciPy states the null as the data being randomly sampled from the same distribution, so a permutation test of a difference in means between groups with unequal spread or shape is not an exact test of equal means, and paired or clustered structure has to be preserved in the reshuffling. Within those conditions the test does not depend on the shape of the distribution, at the price of computation.

Choosing one

The test follows from the question and the data, in that order. The question fixes the parameter, a mean, a rate, a spread, a shape, and whether the alternative is one-sided, such as better than the current process, or two-sided, such as different in either direction. A promise that a new process is at least as good as the old one is a different question again, a non-inferiority question, which needs a pre-specified margin and a null hypothesis of inferiority; failing to reject equality does not establish it. The data fix the rest: numeric values or counts, one group or several, independent groups or matched pairs, and how many observations. The assumptions then have to be checked rather than assumed. For the tests on means the handbook lists three: the data come from a single process with a single distribution, that distribution is normal, and the observations are uncorrelated over time. On the second it is reassuring within limits, since the tests “work fairly well as long as the data are bell-shaped and the tails are not heavy”. The third fails often in analytics, when many rows come from the same user or the same day. Positive correlation within such groups makes the usual standard error too small, and negative correlation can make it too large; renaming the test repairs neither, but an analysis that respects the structure can, whether by aggregating to the unit the question is about, by cluster-robust standard errors, by block resampling, or by a test built on the assignment design. The IID entry covers how to recognize the problem and choose among these. The critical value or p-value that decides the test is only correct if the statistic really has the distribution the test assumes.

What a result does, and does not, tell you

A rejection means the data are unlikely under the null hypothesis at the chosen level; the handbook is blunt that “Statistical significance simply means that we reject the null hypothesis.” It goes on to the consequence that catches most readers: with a very large sample the test may reject the hypothesis that two process means are equal when the difference is too small to matter in engineering terms, and with a small sample a difference that is large in practice may not lead to rejection. Its advice is that “The analyst should not just blindly apply the tests, but should combine engineering judgement with statistical analysis.” The American Statistical Association’s 2016 statement on p-values sets out the same limits as principles. A p-value does not give the probability that the hypothesis is true, nor the probability that the data arose by chance alone. Decisions should not rest on a threshold alone; in the statement’s words, “Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.” And significance says nothing about size: “A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.”

Each of those limits has a companion quantity that supplies what the test leaves out. The size of the effect and its uncertainty come from the estimate and its confidence interval, and whether the size matters is a question of practical significance, not of the test. The chance that a real effect of a given size is missed is \(\beta\), the probability of an error of the second kind; the test’s power is its complement, \(1 - \beta\), the probability of rejecting when that effect is present, and both are probabilities for a fixed effect size, sample size, significance level, and design. The handbook describes the second kind of error directly: “Large discrepancies between reality and the null hypothesis are easier to detect and lead to small errors of the second kind; while small discrepancies are more difficult to detect and lead to large errors of the second kind.” That is what a sample size calculation controls. Running many tests, on many metrics or many segments, raises the number of false rejections in proportion, which is the subject of multiple comparisons; and running the same test repeatedly as data accumulate, stopping at the first rejection, changes the error rate of the procedure, which is the subject of stopping rules. When no formula for the sampling distribution is available, the bootstrap approximates the distribution of an estimator by resampling, and turning that into a test needs a further step, such as centering the data under the null hypothesis or resampling from a model fitted under it; permutation tests build a null distribution directly, but only under the exchangeability described above.

Tests in model monitoring

Statistical tests are also the usual machinery behind drift detection in deployed models: a Kolmogorov–Smirnov test compares the distribution of a numeric feature today with its distribution at training time, a chi-square test does the same for a categorical feature, and the population stability index summarizes the shift as a distance without a formal test. The handbook’s warning about large samples applies with full force here, since at production volumes even small shifts become statistically significant. A test in a monitoring pipeline is therefore best paired with a threshold on the size of the shift, so that an alert means a change large enough to act on rather than one that is merely detectable.

References

• NIST/SEMATECH e-Handbook of Statistical Methods, section 7.1.3 What are statistical tests?
• NIST/SEMATECH e-Handbook of Statistical Methods, section 1.3.5 Quantitative Techniques
• NIST/SEMATECH e-Handbook of Statistical Methods, section 7.1.2 What assumptions are typically made?
• NIST/SEMATECH e-Handbook of Statistical Methods, section 7.3.3 How can we determine whether two processes produce the same proportion of defectives?
• American Statistical Association, press release of 7 March 2016 announcing the ASA Statement on Statistical Significance and P-Values (Wasserstein and Lazar, The American Statistician 70(2), 2016), which reproduces the statement’s six principles
• SciPy documentation, scipy.stats.mannwhitneyu
• SciPy documentation, scipy.stats.permutation_test