Frequentist

Frequentist statistics is a framework in which probability means the long-run behavior of a repeated random experiment or sampling procedure. Model parameters — a population mean, a conversion rate — are treated as fixed but unknown. The data, the estimators and the test statistics are treated as random, because they would come out differently on another sample. Uncertainty is then expressed through the sampling distributions of those estimators under stated model assumptions.

Four principles carry the framework. Parameters are fixed: there is one true value. Samples are random: different samples give different estimates. Procedures are judged by how they behave across hypothetical repetitions. And because they are, they can be designed to control long-run quantities — Type I error rates, confidence interval coverage, power.

The standard tools follow. A point estimate is a single value for an unknown parameter, such as a sample mean. A confidence interval is a range produced by a procedure designed to contain the true parameter at a stated rate across repeated samples, assuming its conditions hold. A hypothesis test evaluates how compatible the data are with a null hypothesis. A p-value is, assuming the null and the other model assumptions hold, the probability of a test statistic at least as extreme as the one observed. Maximum likelihood estimation picks the parameter values making the observed data most likely.

A p-value is not the probability that the null hypothesis is true, and it does not measure the size or the importance of an effect. That sentence is usually where the account stops, and it leaves the obvious follow-up unanswered: if a p-value is not that probability, what does a significant result actually imply?

What a rejection is worth, and what it is conditional on

import numpy as np
from scipy.stats import norm, ttest_ind, t as tdist

print("  a p-value is a statement about data under H0, not about H0.")
print("  What share of significant findings are false depends on how many of")
print("  the hypotheses tested were true in the first place:")
print(f"{'share of true effects':>23}{'power':>8}{'significant results':>21}"
      f"{'of which false':>17}")
alpha = 0.05
for share in (0.01, 0.10, 0.50, 0.90):
    for power in (0.80,):
        tp = share * power
        fp = (1 - share) * alpha
        print(f"{share:>23.2f}{power:>8.2f}{tp + fp:>21.4f}{fp / (tp + fp):>17.4f}")
print(f"{'':>23}{'':>8}{'':>21}")
for power in (0.30, 0.50, 0.80, 0.95):
    share = 0.10
    tp, fp = share * power, (1 - share) * alpha
    print(f"{share:>23.2f}{power:>8.2f}{tp + fp:>21.4f}{fp / (tp + fp):>17.4f}")
print("  at a 5% level with 80% power, a field where one hypothesis in ten is")
print("  true produces significant results that are wrong 36% of the time.")
print("  Nothing here contradicts the 5% - that rate is about the null cases")
print("  only, and it is the mix that decides what a rejection is worth")

rng = np.random.default_rng(0)
runs = 200_000
print(f"\n  and the guarantee is conditional on the model. Two-sample t-tests")
print(f"  with no real difference, at a nominal 0.05:")
print(f"{'n1':>6}{'n2':>6}{'sd1':>7}{'sd2':>7}{'Student':>11}{'Welch':>11}")
for n1, n2, s1, s2 in ((30, 30, 1.0, 1.0), (30, 30, 1.0, 3.0),
                       (10, 50, 1.0, 3.0), (50, 10, 1.0, 3.0)):
    a = s1 * norm.ppf(rng.random((runs, n1)))
    b = s2 * norm.ppf(rng.random((runs, n2)))
    p_eq = ttest_ind(a, b, axis=1, equal_var=True).pvalue
    p_w = ttest_ind(a, b, axis=1, equal_var=False).pvalue
    print(f"{n1:>6}{n2:>6}{s1:>7.1f}{s2:>7.1f}{np.mean(p_eq < 0.05):>11.4f}"
          f"{np.mean(p_w < 0.05):>11.4f}")
print("  equal-size tests are near nominal in this normal simulation.")
print("  Unequal sizes with unequal variances distort this pooled test: when the")
print("  larger group carries the larger spread it rejects a true null 0.0006")
print("  of the time, and when the smaller group carries it, 0.2918. Welch")
print("  stays near 0.05 throughout, which is why it is the sensible default")

print(f"\n  what the framework does deliver, when its conditions hold:")
mu, sd, n = 0.0, 1.0, 40
runs = 200_000
x = mu + sd * norm.ppf(rng.random((runs, n)))
m, s = x.mean(axis=1), x.std(axis=1, ddof=1)
half = tdist.ppf(0.975, n - 1) * s / np.sqrt(n)
print(f"    95% interval coverage over {runs:,} samples   "
      f"{np.mean(np.abs(m - mu) <= half):.4f}")
print(f"    share of p-values below 0.05 under H0        "
      f"{np.mean(2 * tdist.sf(np.abs(m) / (s / np.sqrt(n)), n - 1) < 0.05):.4f}")
print(f"    share below 0.01                             "
      f"{np.mean(2 * tdist.sf(np.abs(m) / (s / np.sqrt(n)), n - 1) < 0.01):.4f}")
print("  the long-run rates are exactly what they claim. That is the whole")
print("  offer - a procedure with a known failure rate across repetitions,")
print("  and no statement at all about the particular result in front of you")
#   a p-value is a statement about data under H0, not about H0.
#   What share of significant findings are false depends on how many of
#   the hypotheses tested were true in the first place:
#   share of true effects   power  significant results   of which false
#                    0.01    0.80               0.0575           0.8609
#                    0.10    0.80               0.1250           0.3600
#                    0.50    0.80               0.4250           0.0588
#                    0.90    0.80               0.7250           0.0069
#
#                    0.10    0.30               0.0750           0.6000
#                    0.10    0.50               0.0950           0.4737
#                    0.10    0.80               0.1250           0.3600
#                    0.10    0.95               0.1400           0.3214
#   at a 5% level with 80% power, a field where one hypothesis in ten is
#   true produces significant results that are wrong 36% of the time.
#   Nothing here contradicts the 5% - that rate is about the null cases
#   only, and it is the mix that decides what a rejection is worth
#
#   and the guarantee is conditional on the model. Two-sample t-tests
#   with no real difference, at a nominal 0.05:
#     n1    n2    sd1    sd2    Student      Welch
#     30    30    1.0    1.0     0.0495     0.0493
#     30    30    1.0    3.0     0.0531     0.0503
#     10    50    1.0    3.0     0.0006     0.0494
#     50    10    1.0    3.0     0.2918     0.0506
#   equal-size tests are near nominal in this normal simulation.
#   Unequal sizes with unequal variances distort this pooled test: when the
#   larger group carries the larger spread it rejects a true null 0.0006
#   of the time, and when the smaller group carries it, 0.2918. Welch
#   stays near 0.05 throughout, which is why it is the sensible default
#
#   what the framework does deliver, when its conditions hold:
#     95% interval coverage over 200,000 samples   0.9491
#     share of p-values below 0.05 under H0        0.0509
#     share below 0.01                             0.0101
#   the long-run rates are exactly what they claim. That is the whole
#   offer - a procedure with a known failure rate across repetitions,
#   and no statement at all about the particular result in front of you

Three readings

The first table is a frequentist calculation throughout, and it answers the follow-up. At a 5% level with 80% power, a research area where one hypothesis in ten is true produces significant results that are wrong 36% of the time. Where one in a hundred is true, 86% of them are wrong. Nothing here contradicts the 5%: that rate applies to the null cases only, and it says nothing about how many of the tested hypotheses were null. Power moves the number too — at 30% power, six of every ten significant results in the same area are false. This is why underpowered studies are a problem for the literature and not only for the study.

The second table is the other half of the answer. The guarantee holds conditional on the model. In this normal simulation with 30 observations per arm, the pooled t-test stays close to the nominal rate despite unequal variances; equal sizes alone do not guarantee finite-sample validity under arbitrary heteroscedasticity. With unequal sizes and unequal variances here it fails in both directions: when the larger group carries the larger spread it rejects a true null 0.0006 of the time, and when the smaller group carries it, 0.2918. Welch’s version stays near 0.05 in every row, which is why it is the sensible default rather than a correction to reach for after a variance test.

The third confirms what the framework does deliver when its conditions hold. Coverage of 0.9491 against a nominal 0.95, and p-values below 0.05 and 0.01 at 0.0509 and 0.0101 under the null. That is the whole offer: a procedure with a known failure rate across repetitions, and no statement about the particular result in front of you.

A/B testing, as the standard case

Comparing two versions of a page is the everyday form. A two-sided test states \(H_0: p_A = p_B\) against \(H_1: p_A \neq p_B\), and a p-value below a pre-selected level such as \(\alpha = 0.05\) is evidence against the null, on which the null may be rejected.

It does not prove the versions differ, nor that B is better. The estimated difference, its confidence interval, the test’s assumptions, the sample size and the practical importance all belong in the reading — and so does the first table above, since the answer to “how often is a significant A/B result wrong” depends on how often the changes being tested do anything at all. If the question is specifically whether B increases conversions, a one-sided alternative is appropriate, provided it was chosen before the results were seen.

Against the Bayesian framework

The two differ on what probability describes. Here it describes long-run frequencies, or the repeated-sampling behavior of a procedure; in Bayesian inference it quantifies uncertainty conditional on a model. Parameters are fixed but unknown here, and represented by distributions there. Data are random under the assumed sampling process here, and fixed once observed there. Prior information is not carried by a prior distribution here, and is there. The outputs are point estimates, confidence intervals, p-values and tests against posteriors, credible intervals and posterior probabilities.

The difference shows most plainly in how a 95% interval reads. A procedure producing 95% confidence intervals contains the true fixed parameter in 95% of repeated samples under its assumptions. A 95% credible interval says that, given the prior, likelihood, model and data, the parameter lies in it with probability 0.95. And in sequential settings: repeatedly checking a conventional fixed-sample test inflates the Type I error rate unless an appropriate sequential method is used, while Bayesian updating accommodates data as it arrives — though the validity and performance of optional stopping still depend on the model, the prior, the stopping rule and the inferential goal.

Strengths and limits

The framework offers established methods for controlling long-run error rates, a mature toolkit for experiments, clinical trials, surveys and A/B testing, no requirement to specify a prior, and well-developed theory for estimation, testing, model comparison and design.

Its limits are the mirror image, and the measurements above put numbers on three of them. P-values and confidence intervals are widely misread — the first table shows what the correct reading leaves open. A statistically significant result may have no practical importance. Results depend on model assumptions, sampling design and stopping rules — the second table shows one assumption failing by a factor of six. And conventional methods do not supply the probability that a hypothesis is true, which is precisely the quantity the first table had to construct a prior share for.

References

• SciPy documentation, scipy.stats.ttest_ind and scipy.stats.t. docs.scipy.org