Statistical Significance
Statistical significance says how incompatible the data are with a null model at a chosen threshold. It does not say how large or how important an effect is.
-
-
Z-Score
An obvious outlier among ten values scores 2.8315, which is 99.5% of the maximum that sample size allows.
-
Two-Proportion Z-Test
Do two groups convert at the same rate? The pooled z statistic, a worked A/B example with its interval, when to use Fisher’s exact test, and the A/B pitfalls.
-
Minimum Detectable Lift (MDL)
A 12% lift at target power does not rule out detecting 2%. State the power, effect units, and design before judging whether the experiment is worthwhile.
-
Trivial Effects
Significance says a difference is detectable. Whether it is worth anything is a separate question that no test answers.
-
Sample size
Ten thousand events from two thousand users is not n = 10,000 for a question about users. The unit you sampled decides what n is.
-
Power (1 – β)
Power is the probability of rejecting the null when a specific alternative is true. Effect size, sample size, α, and variability set it, and it changes with the test design.
-
Significance Level (α)
α is a tolerance you choose, not a measure of evidence. And the risk it trades against goes up as you push it down.
-
Effect Size (δ)
Divide by the standard deviation and you have an effect size. Divide by the standard error and you have a test statistic. They are not the same number.
-
Hypothesis Testing
A test rejects or fails to reject — it never confirms. And “fail to reject” covers both no effect and not enough data to tell.
-
Causal Impact
The estimate is the gap between what happened and a forecast of what would have happened. Everything depends on that forecast being right.
-
A/B/n Test
The winning arm is the maximum of several noisy estimates, so its measured lift is biased upward by construction.
-
Multivariate Test (MVT)
Testing several page elements in one experiment. What a factorial design buys, what it does not cost, and how the arm count multiplies the false positives.
-
Risk of Peeking
Daily monitoring need not change inference. Stopping on a fixed-sample p-value does; valid sequential methods account for repeated looks and stopping.
-
Causal Inference
Causal inference from observational data needs data and identifiability conditions. Choosing a method does not establish the identification assumptions.
-
P-Value (probability value)
A p-value is a probability about data given a model. Reading it as a probability about the hypothesis inverts the conditional.
-
Z-Test
Estimate minus null value, over a known standard error, against the normal curve. Exact for normal data with known sigma, else approximate; mostly proportions.
-
T-Test
Estimate over its estimated standard error, referred to Student’s t. One-sample, paired, and two-sample forms, when to use Welch, and heavy tails.
-
Statistically Significant
“Unlikely due to chance” is the one reading a p-value cannot support. What it measures is incompatibility with a model, assumptions included.
-
IID (Independent and Identically Distributed)
Same distribution, no dependence between draws. The assumption behind the usual standard errors and intervals, and the one grouped or time-ordered data breaks.
-
Temporal autocorrelation (Serial Correlation)
Serial dependence changes the information in a time series. Positive and negative dependence can affect uncertainty in opposite directions.
-
Cramér’s V
Twenty categories over two hundred rows, independent by construction, average a V of 0.3077.
-
Treatment Cost
What applying an intervention costs per unit — including the margin given away to customers who needed no persuading.
-
Incremental Revenue
The revenue a treatment caused rather than accompanied. Heavy-tailed, wide-intervalled, and not profit until a margin and a cost are applied.
-
Incremental Recovery Rate (IRR)
How much more recovery an intervention produced than doing nothing — a rate, in percentage points, and not a quantity until a population is attached.
-
Incremental Sales
The units a campaign actually caused. Why counting them is not the same as counting revenue, and why a promotion can win on one and lose on the other.
-
Random Targeting Strategy
Random assignment supports the plain treated-versus-control comparison; random ranking supplies an uplift baseline. Observational estimates need additional identification and adjustment assumptions.
-
Causal ML (Causal Machine Learning)
Causal ML relaxes the shape of the model, not the assumptions that make a comparison causal. Those stay exactly as strong as before.
-
Cumulative Uplift
The incremental outcome accumulated as targeting goes deeper down an uplift ranking. Why it cannot be a sum over customers, and how to read where it flattens.
-
Population Proportion
Count every student and the textbook formula still reports plus or minus 0.96 percentage points.
-
Incremental Gain
The extra outcome produced inside one slice of a ranking. Computed from response rates rather than counts, and not additive across slices when the treated share differs between them.
-
Total Incremental Benefit (TIB)
Everything a campaign produced that would not have happened anyway. Converting it to money, carrying its interval, and why treating everyone rarely wins.
-
Cumulative Incremental Gain (CIG)
The incremental outcome collected down to a depth of a ranking. Accumulating slice by slice gives a different answer from computing it directly.
-
Qini Curve
Radcliffe’s incremental gains chart: treated responders minus what an equal treated group would have produced untreated. Its baselines, and why they matter.
-
Uplift Score
The per-unit output of an uplift model: a difference of two predicted probabilities, not a probability. What it supports, and what it cannot be checked against.
-
Uplift Models
Uplift models estimate conditional treatment effects. Targeting also needs outcome values, action costs, constraints, and uncertainty.
-
Demand Forecasting
Customer demand never changed. Four stages later the factory plans against a signal 43.8 times as volatile.
-
Prophet Forecasting
Prophet offers configurable forecasting at scale. Two FPP3 examples favor other methods on accuracy; assess its performance on your own series and forecast horizon.
-
ARIMA (AutoRegressive Integrated Moving Average)
The d in ARIMA(p,d,q) decides the shape of your long-horizon uncertainty. With d = 0 intervals converge; with d ≥ 1 they keep growing.
-
Return Distribution
Plus 10%, minus 10%, twice: the arithmetic mean is zero and 98.01% of the money is left.
-
Value-at-Risk (VaR)
VaR is a quantile of a loss distribution. It marks where the tail starts and is silent about what happens beyond that point.
-
Risk Forecast
“How likely is demand above capacity?” is a question about a region of the distribution. A point forecast has no way to answer it.
-
Probabilistic Scoring
Three forecasters saying 0.70, 0.80 and 0.99 are correct on exactly the same 70% of days.
-
Full Distribution Forecast
No finite set of quantiles determines a mean. A mean is an integral over the whole line.
-
Continuous Probabilistic Forecasting
Fit a Normal to data from a t and the probability of a loss beyond 5% comes out 0.001653 against a truth of 0.005862.
-
Quantile Forecasts
Asymmetric costs have an asymmetric answer. If being short costs three times as much as being long, the target is not the mean.
-
Point Forecasts
A point forecast is one summary of a distribution. Reporting only the summary discards the part a decision usually needs.
-
Probability Forecasts
Days it calls 0.0152 happen 23.81% of the time, and accuracy ranks it level with the truth.
-
Probability Density
The same heights give f = 0.039894 per cm and 3.989423 per meter. The interval probability is 0.682689 either way.
-
Normal Distribution
Most forecast intervals assume normality, and the assumption is least reliable exactly in the tails — which is where risk questions live.