Continuous Ranked Probability Score (CRPS)
CRPS reads like a weighted absolute error computed from the entire distribution, so it rewards being both accurate and appropriately confident.
-
-
MASE (Mean Absolute Scaled Error)
A forecast’s MAE divided by the training-set error of a naive forecast. Unit-free and comparable across series, and the denominator is easy to get wrong.
-
Pinball Loss (a.k.a. Quantile Loss)
At p = 0.5 it is just absolute error. Away from the median it weights misses unevenly, which is what makes a quantile the thing being estimated.
-
WMAPE (Weighted Mean Absolute Percentage Error)
WMAPE needs an explicit formula. Actual weights can reproduce WAPE, while business weights, aggregation, and zero handling change the result.
-
sMAPE (Symmetric Mean Absolute Percentage Error)
The same ten-unit miss scores 0.09524 high and 0.10526 low. The name describes an intention, not the formula.
-
RMSLE (Root Mean Squared Logarithmic Error)
A doubled forecast contributes 0.480384 or 0.164402 to MSLE at actuals 10,000 and 1. RMSLE takes the square root of the mean contribution.
-
Mean Absolute Error (MAE)
The average absolute gap between forecast and actual, in the target’s own units. Easy to read, linear in the error, and a forecast of the median when minimized.
-
WAPE (Weighted Absolute Percentage Error)
Two forecasts with identical 8% WAPE: one misses the big item by 11.4%, the other forecasts a small item at a fifth of its demand.
-
Uplift
The difference between the outcome with the treatment and the outcome without it. Why a response model does not estimate it, and what the number is attached to.
-
Mean Absolute Percentage Error (MAPE)
Error as a percentage of the actual, averaged. Comparable across series, but it breaks at zero, explodes near it, needs a real zero, and rewards low forecasts.
-
Root Mean Squared Error (RMSE)
Square the errors, average, take the root: same units as the target, but large misses count more. Minimizing it forecasts the mean; it is never below MAE.
-
Noninferiority Margin
A noninferiority test asks whether a new option is worse than the old one by no more than a margin chosen in advance, and it passes only when the confidence interval’s lower bound clears that margin.
-
Sample Ratio Mismatch
A sample ratio mismatch is an alarm about experiment counts. It prompts investigation of assignment, exposure, logging, and filtering before outcome differences are trusted.
-
Rule of Three
Zero failures do not imply a zero failure rate. The rule of three gives an approximate upper bound, while the exact formula shows how the approximation behaves at small sample sizes.
-
Stratified Sampling
Sampling more rare cases helps estimate their performance, but an overall population rate still needs the population weights. A worked example separates those two goals.
-
Rogan–Gladen Correction
An imperfect judge catches some real failures and flags some good cases. Rogan–Gladen correction separates those contributions to estimate an aggregate failure rate.
-
McNemar Test
Compare two systems on the same cases by counting where only one passes. The exact test and worked counts explain why pairing changes the evidence for a difference.
-
Bradley–Terry Model
The Bradley–Terry model turns pairwise preferences into relative strength scores. Its probability formula, fitting example, and failure cases explain what a ranking can support.
-
Winner’s Curse
Selecting a candidate by its measured score can select favorable noise along with quality. Simulated candidate searches illustrate optimism and the role of held-out confirmation.
-
Design Effect
Related items contribute less independent information than their row count suggests. The design effect connects cluster size and correlation to variance and effective sample size.
-
Paired Bootstrap
Resample the same test items for both systems to estimate uncertainty in their score difference. A worked example compares paired and unpaired intervals.
-
Good–Turing Estimator
Singleton counts help estimate the probability mass of outcomes absent from a sample. See the adjusted-count calculation and why sparse frequency tables need smoothing.
-
Pre-Registered Decision Gate
Write down what will be measured, what counts as enough, and what happens in each outcome before the pilot starts. The rule then limits how the result can be read.
-
Intention-to-Treat Analysis
In a randomized offer, only some people take it up, and those who do are not typical. Intention-to-treat compares everyone as assigned, which keeps the randomization and estimates the effect of the offer.
-
Difference-in-Differences
Compare the change in a treated group with the change in a comparison group. The method removes a permanent gap and a shared trend, and relies on the groups having moved in parallel.
-
Poisson Process
Learn the model behind counting events in time, why its variance equals its mean, and which real event streams violate the independence it assumes.
-
Oversmoothing and Undersmoothing
Learn how to tell a curve that has erased real structure from one that is tracing noise, and why every smoothing choice trades one for the other.
-
Mean Integrated Squared Error
Learn how MISE measures the gap between an estimated curve and the unknown truth, and why it cannot be computed directly but can still be minimized.
-
Histogram Bin Width
Learn what the default bin rules assume, why bin width is an analytical choice rather than a display setting, and how to pick it from data.
-
Percentile
Understand percentiles through ordered observations, with p95 examples, calculation conventions, and aggregation limits.