Reading a Result Honestly: The Statistical Judgment Behind Every Number

A dashboard says conversion rose 3.2% after the redesign. The number is correct — the query is right, the data is complete, nobody made a mistake. Whether it means the redesign worked is a separate question, and answering it requires judgments that no amount of care in the calculation supplies.

This is not about doing statistics. It is about reading results, which is what most people in an organization actually do with analytical work, and about the handful of questions that separate a number you can act on from one you cannot.

Every number is a sample of something

The instinct that having all the data removes sampling concerns is one of the most consequential misreadings in business analytics. Complete data about last month describes last month exactly — as a piece of description it is not a sample of anything, and treating it as one is its own confusion. The moment it is used to say something about a wider population or about what comes next, it becomes a sample — of the population you care about, which is future months, or all customers rather than the ones who used the product, or the general case rather than this particular period.

So the question to ask of any figure is not how much data it rests on but what it is meant to represent. Three variants of the mismatch account for most trouble.

• Time. Last quarter’s measurement describes last quarter. Treating it as the current rate assumes stability, and forecasting texts are explicit that predictability depends partly on “how similar the future is to the past” — an assumption, not a property of the data.
• Coverage. A satisfaction score computed from responses describes respondents, and selection and nonresponse bias is the gap between them and everyone else. Complete data on a self-selected group is not complete data.
• Unit. A rate computed per session answers a different question from the same rate per user, and switching between them mid-argument is an easy way to reach whichever conclusion you started with.

None of this argues against using the data. It argues for writing down, next to the number, what population it describes — which takes one sentence and prevents most downstream overreach. Where the figure genuinely comes from a subset, the mechanics of statistical sampling determine what it supports, and a convenience sample supports much less than its size suggests.

Uncertainty is a range, and the range is the result

A single number implies a precision it does not have. Two measurements of the same thing on different weeks differ, and some of that difference is the thing changing while the rest is noise. An interval does not split an observed difference into those two parts — nothing does that from the numbers alone. What it states is how much uncertainty the stated statistical model attributes to an estimate, or to a future observation. That is enough to tell you whether the difference you are looking at is larger than the uncertainty, which is the question worth asking.

Forecasting has the cleanest statement of why this matters: “If we only produce point forecasts, there is no way of telling how accurate the forecasts are.” A prediction interval “gives an interval within which we expect y_t to lie with a specified probability,” and it behaves in a way worth internalizing — intervals “usually increase in length as the forecast horizon increases,” because “the further ahead we forecast, the more uncertainty is associated with the forecast.” A twelve-month forecast presented with the same apparent precision as a one-month forecast is misrepresenting what is known.

For a measured quantity rather than a forecast, the counterpart is a confidence interval, built from the standard error — counterpart rather than equivalent, because the two intervals are about different things. A prediction interval is about where a future observation will fall. A confidence interval is about an unknown quantity you are estimating, and its stated level is a property of the procedure across repeated samples rather than a probability attached to this one interval. Conflating them makes a confidence interval sound like a forecast of individual values, which is much wider than what it claims. Three questions still get most of the value out of either without any calculation.

• Does the interval include a value that would change the decision? An estimate of 3.2% with a range from −1% to 7% and a break-even at 2% has not settled anything.
• How wide is it relative to the effect? A range several times the size of the estimate is a statement that the measurement is not yet informative, whatever the midpoint says.
• What does it not include? A confidence interval reflects sampling variation in the estimate; a prediction interval also includes the variability of an individual future observation. Neither reflects anything the model does not represent — a mislabelled event, a tracking change mid-period, a population that shifted, or a bias in how the data was collected. Those sources are usually larger, and they are invisible in the interval.

That last point deserves emphasis because it inverts the usual intuition. Statistical uncertainty is the part we can quantify, which makes it the part that gets reported, which makes it look like the main risk. In most business analysis it is not.

Bias is not an error in the calculation

Noise averages out with more data; bias does not, and more data makes a biased estimate more precisely wrong. Three kinds show up constantly, and each is diagnosed by asking about the collection process rather than the numbers.

Kind	How it enters	The question that finds it
Selection	Who ends up in the data is related to what you are measuring — respondents, opt-ins, users who reached the page	What had to happen for a case to appear here, and is that related to the outcome?
Survivorship	The cases that ended are missing. Retention measured on current customers omits everyone who left — see survivorship bias	Who is absent from this dataset, and why did they leave it?
Measurement	The instrument changed or was never neutral: a new SDK version, a renamed event, a question that suggests its answer	Did anything about how this was recorded change during the period being compared?

The measurement row is the one that quietly ruins period comparisons. A tracking change deployed in week three makes weeks one and two incomparable with weeks four onward, and nothing in the resulting chart indicates it. Asking when the instrumentation last changed should be as routine as asking what the number is.

The more you look, the more you find

If you compare twenty segments, some will look different by chance alone. That is not a flaw in anyone’s method; it is arithmetic, and it is the mechanism behind multiple comparisons problems. Two organizational habits create it without anyone intending to.

• Slicing until something appears. A dashboard that lets anyone break a metric down by twelve dimensions is a machine for producing apparently meaningful differences, and the ones that get reported upward are the extreme ones by construction.
• Checking a running experiment repeatedly. Stopping when the result looks good makes the stopping rule part of the analysis, and a test evaluated continuously will cross a threshold eventually whether or not there is an effect.

The American Statistical Association’s statement on p-values names the general remedy as one of its six principles: “Proper inference requires full reporting and transparency.” In practice that means saying how many comparisons were made and which analyses were decided in advance — which is why experiment design asks for the metric, the segments, and the duration to be fixed before the test starts. A finding from an unplanned slice is a hypothesis, and the honest next step is to test it rather than to report it.

Significant does not mean large, and does not mean true

The ASA’s six principles are worth reading in full by anyone who reads test results, because they contradict the way the word is used in most meetings. The statement defines a p-value as “the probability under a specified statistical model that a statistical summary of the data (for example, the sample mean difference between two compared groups) would be equal to or more extreme than its observed value,” and then draws the boundaries explicitly.

• “P-values can indicate how incompatible the data are with a specific statistical model.”
• “P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.”
• “Scientific conclusions and business or policy decisions should not be based only on whether a P-value passes a specific threshold.”
• “A P-value, or statistical significance, does not measure the size of an effect or the importance of the result.”
• “By itself, a P-value does not provide a good measure of evidence regarding a model or hypothesis.”

Read together with “Proper inference requires full reporting and transparency,” those principles rule out the two most common readings of a test result: that statistical significance establishes the effect is real, and that it establishes the effect is worth having. The p-value is one input, and a large sample will make a trivial difference significant while a small one will leave a large difference undetected — which is what statistical power is about.

Which is why the useful pairing is the effect size and its interval against a threshold set in advance — the difference that would actually change the decision. That comparison is practical significance, and reporting it alongside the statistical result prevents both failure modes at once: celebrating a 0.1% lift because it cleared a threshold, and discarding a promising 4% lift because the test was underpowered.

What experiments answer, and what observation answers

The strongest claim available from data depends on how it arose, and conflating the two is how an analysis becomes an unjustified recommendation.

	Randomized experiment	Observational data
What it supports	The effect of the change you made, on the population you tested, during the period you ran it	An association, plus a causal estimate if you state assumptions about what else differed
Main threat	Generalizing beyond the tested population, period, or version of the change	Something you did not measure explaining the difference
What improves it	Longer runs, pre-registered metrics, replication in another context	Explicit assumptions, sensitivity analysis, and an honest statement of what would break the conclusion

Two clarifications are worth making, because the table invites two wrong conclusions. First, observational data is not disqualified from causal claims — causal inference is a developed methodology, and as Hernán puts it, “causal inference is a core task of science, regardless of whether the study is randomized or nonrandomized.” What it requires is that the assumptions be stated rather than assumed away, and they cannot be checked against the same data. Relatedly, method choice is not interchangeable across purposes: “Automatic variable selection procedures may work for prediction, but not necessarily for causal inference.”

Second, an experimental result is narrower than it feels. The rules of thumb Kohavi and colleagues generalized from experiments at “Amazon, Booking.com, LinkedIn, and multiple Microsoft properties” illustrate the shape of the knowledge well: “Speed matters” is a real and repeatedly observed effect, and it comes with the qualification that “certain areas of the web page are more critical.” An effect established in one place, on one audience, in one implementation, is evidence about that combination and a hypothesis about others.

Timing adds one more caution, worth stating because it is easy to miss: a change can produce a short-lived response simply because it is new, and a measurement taken in the first days may capture curiosity rather than value. Where a result matters, the answer is to plan the observation period rather than to add a fixed amount of time: decide it from the business cycle the metric follows, how often users encounter the change, and the sample the comparison needs. What settles it is looking at the effect over time rather than at one total — a lift that decays across weeks is a different finding from one that holds. Kohavi and colleagues treat this directly under novelty and primacy effects, and their own guidance is a duration recommendation for that check rather than a universal waiting period.

How to report a result you can defend

Everything above collapses into a short reporting habit. Five lines, none of them requiring new analysis.

• The estimate and its range, not the estimate alone.
• What population and period it describes, in one sentence.
• How it arose — a planned experiment, a pre-specified analysis, or an exploration that found something.
• The comparison that matters — the effect against the threshold that would change the decision.
• What would make this wrong — the assumption most likely to fail, or the confounder you could not rule out.

The last line is the one that builds trust, and it is the one most often left out for fear of undermining the work. It does the opposite: an analysis that names its own weak point is much harder to dismiss later, and a reader who is told what would break the conclusion can judge whether that thing is plausible in their part of the business. Correlation presented as correlation, with its limits stated, is more useful than a causal claim nobody can check.

References: Wasserstein & Lazar, The ASA’s Statement on p-Values: Context, Process, and Purpose, The American Statistician 70:129–133, 2016 (six principles as reproduced in Table I of this open-access discussion); Hyndman & Athanasopoulos, Forecasting: Principles and Practice (3rd ed.), §5.5 Prediction intervals; Hernán, The C-Word, American Journal of Public Health 108(5), 2018; Kohavi, Deng, Longbotham & Xu, Seven Rules of Thumb for Web Site Experimenters, KDD 2014.