Subsampling
Asking sklearn.utils.resample for 2,000 rows returned 1,890 distinct ones. The default is replace=True, which is a bootstrap, not a subsample.
-
-
Standard Error (SE)
Standard error describes sampling variability; a margin of error also uses a confidence level and reference distribution. A normal 95% interval uses about 1.96 times the SE.
-
Margin of Error (MoE)
A margin of error needs a confidence level to exist. Quote one without it and you are off by the multiplier — about a factor of two at 95%.
-
IID (Independent and Identically Distributed)
Same distribution, no dependence between draws. The assumption behind the usual standard errors and intervals, and the one grouped or time-ordered data breaks.
-
Active Learning
From a 10-label seed, uncertainty sampling reached 0.5714 and random sampling 0.6418 on the same budget. It only won once the seed reached 400 in this experiment.
-
Population Proportion
Count every student and the textbook formula still reports plus or minus 0.96 percentage points.
-
Stratified Cross-Validation
With 12 positives in 500 rows, plain five-fold produced an empty fold in 56 of 200 shuffles. Stratification avoids that but changes split variability.
-
Reproducible Test Sets for AI Evaluation
A 27-versus-25 score on 30 pasted questions proves little. Build a test set from a failure taxonomy, sample it by stratum, score with checks and validated judges, and compare releases pairwise.
-
Stratified Sampling
Sampling more rare cases helps estimate their performance, but an overall population rate still needs the population weights. A worked example separates those two goals.
-
Good–Turing Estimator
Singleton counts help estimate the probability mass of outcomes absent from a sample. See the adjusted-count calculation and why sparse frequency tables need smoothing.
-
Head and Tail Sampling
Head sampling decides whether to keep a trace before it has run; tail sampling decides after it has collected the trace’s spans for a bounded wait. The choice determines which failures you can find and how far you can trust the rates you compute.
-
Sampling
Understand sampling in language generation, with temperature, top-k, top-p, and the limits of reproducibility.