P-Value (probability value)

A p-value is the probability of observing data at least as extreme as what you observed, computed under a specified statistical model — the null hypothesis together with the test’s assumptions. It is a number about data, conditional on a model.

Every widespread misreading comes from moving the conditional. The American Statistical Association’s 2016 statement was written largely to stop that, and its six principles are the backbone of this entry.

What it is

Principle 1 states the capability: “p-values can indicate how incompatible the data are with a specified statistical model.” Read the object carefully — a model, not a hypothesis. The model includes how the data were assumed to be sampled, whether observations are independent, and what distribution the statistic follows.

So a small p-value is evidence against the whole package, and a violated assumption produces one just as a real effect does. That is not a caveat; it is what the principle says.

What it is not

Principle 2 rules out the two readings people reach for: “p-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.”

Quantity	Direction of the conditional	Does a test give it?
p-value	P(data this extreme given the model)	Yes — this is the output
Probability the hypothesis is true	P(hypothesis given the data)	No. Requires a prior over hypotheses
“Probability this is a fluke”	Same inversion, informally worded	No

The gap between rows one and two is not a technicality that careful wording bridges. Getting from one to the other needs something the test never supplies, which is why “there is only a 5% chance this is a fluke” cannot be derived from \(p = 0.05\) at all. The phrase to retire is “unlikely due to chance,” because it invites exactly that reading — see statistically significant.

Principle 5 removes a third reading: the value “does not measure the size of an effect or the importance of a result.” A p-value has no units and does not, by itself, determine an effect’s magnitude — that is effect size.

Why it moves with the sample size

The arithmetic is worth seeing once, because it explains a whole class of misleading results. A test statistic is a difference divided by its standard error, and the standard error shrinks like \(1/\sqrt{n}\). So for a fixed real difference, the statistic grows as \(\sqrt{n}\) and the p-value falls.

With enough data almost any nonzero difference clears any threshold. A 0.05% change measured across ten million users can produce \(p < 0.001\) and still be worthless. In the other direction a small study can leave a large real effect unresolved, which is why a large p-value is not evidence of absence — see statistical power and sample size.

The threshold it gets compared against is a separate choice with its own consequences: see significance level and Type I error. Whether a detected difference is large enough to matter is not a statistical question at all — see trivial effects.

How it stops meaning what it says

A p-value is computed against a reference procedure, and an unadjusted fixed-sample p-value stops describing the analysis once the data decide what gets reported or when to stop. Run twenty comparisons at \(\alpha = 0.05\) and report the one that cleared, and the reported number no longer describes the procedure that produced it. The ASA names the practice — “‘p-hacking’ and ‘data dredging’ that emphasize the search for small p-values over other statistical and scientific reasoning” — and its fourth principle gives the remedy: “proper inference requires full reporting and transparency.”

With a plain fixed-sample p-value, that means fixing the metric, the population, and the stopping rule before looking, and reporting what you ran rather than what worked. Multiplicity adjustments and sequential or anytime-valid methods build the selection or stopping into the reference procedure instead. Repeatedly checking an accumulating result is the same failure in time rather than in breadth — see risk of peeking.

What to report beside it

Principle 6 is the one to hold on to: “by itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.” Not that it is worthless — that it is uninformative alone.

The ASA lists what statisticians use to supplement or replace it, and the first group is the one most decisions want: methods “that emphasize estimation over testing such as confidence, credibility, or prediction intervals; Bayesian methods; alternative measures of evidence such as likelihood ratios or Bayes factors; and other approaches such as decision-theoretic modeling and false discovery rates.” An interval answers how big and how precisely known in one object — see confidence intervals — which is most of what a reader of a result actually needs.

References

• American Statistical Association, statement on statistical significance and p-values (March 7, 2016). The six principles are quoted from that statement.