Interpreting Analytical Results
The question is not whether a result is significant. It is what the sample represents, how wide the uncertainty is, and whether the difference matters.
-
-
Statistical Sampling
A larger convenience sample is a more precise measurement of the wrong group. Size does not repair how the rows were selected.
-
Practical Significance
A big enough sample makes a 0.1% difference significant. Whether 0.1% is worth the engineering is a different question, decided in advance.
-
Survivorship Bias
A current-status filter can hide former members. Recovering a historical population depends on retained membership history and explicit missing-data checks.
-
Multiple Comparisons
Slicing a metric until something stands out is a search procedure. Selected findings need error control that accounts for the search.
-
Statistical Significance
Statistical significance says how incompatible the data are with a null model at a chosen threshold. It does not say how large or how important an effect is.
-
Power (1 – β)
Power is the probability of rejecting the null when a specific alternative is true. Effect size, sample size, α, and variability set it, and it changes with the test design.
-
Harmonic Mean
Precision 0.99 with recall 0.02 gives an arithmetic mean of 0.5050 and a harmonic mean of 0.0392. The large number is all but ignored.
-
Sample Ratio Mismatch
A sample ratio mismatch is an alarm about experiment counts. It prompts investigation of assignment, exposure, logging, and filtering before outcome differences are trusted.
-
Rule of Three
Zero failures do not imply a zero failure rate. The rule of three gives an approximate upper bound, while the exact formula shows how the approximation behaves at small sample sizes.
-
Stratified Sampling
Sampling more rare cases helps estimate their performance, but an overall population rate still needs the population weights. A worked example separates those two goals.
-
Rogan–Gladen Correction
An imperfect judge catches some real failures and flags some good cases. Rogan–Gladen correction separates those contributions to estimate an aggregate failure rate.
-
McNemar Test
Compare two systems on the same cases by counting where only one passes. The exact test and worked counts explain why pairing changes the evidence for a difference.
-
Bradley–Terry Model
The Bradley–Terry model turns pairwise preferences into relative strength scores. Its probability formula, fitting example, and failure cases explain what a ranking can support.
-
Winner’s Curse
Selecting a candidate by its measured score can select favorable noise along with quality. Simulated candidate searches illustrate optimism and the role of held-out confirmation.
-
Design Effect
Related items contribute less independent information than their row count suggests. The design effect connects cluster size and correlation to variance and effective sample size.
-
Paired Bootstrap
Resample the same test items for both systems to estimate uncertainty in their score difference. A worked example compares paired and unpaired intervals.
-
Good–Turing Estimator
Singleton counts help estimate the probability mass of outcomes absent from a sample. See the adjusted-count calculation and why sparse frequency tables need smoothing.
-
Weighted Average
Learn when a plain average misleads, how weights change the answer, and why averaging group averages is the most common version of the mistake.
-
Percentile
Understand percentiles through ordered observations, with p95 examples, calculation conventions, and aggregation limits.