Inter-Annotator Agreement
Two reviewers agree on 85% of labels. Kappa adds a marginal-frequency comparison, but neither measure establishes correctness or a ceiling for model judges.
-
-
Feedback Loop (Machine Learning)
A model can change which outcomes are observed and the outcomes themselves. Correcting the resulting bias depends on the target, coverage, and policy.
-
Holdout Reuse
A test set stops being an independent final check when its results guide model selection. One look per candidate does not prevent that feedback.
-
Evaluation Protocol
An evaluation protocol records the task, data boundaries, decision rules, and uncertainty methods before results influence model selection.
-
Class Imbalance
When fraud is one case in a thousand, a model that never flags fraud is 99.9% accurate. Imbalance is a fact to evaluate around, not a defect to remove.
-
ML Problem Framing
A model can be accurate and useless if it predicts the wrong thing. Framing decides what is predicted, for which decision, and how success is judged.
-
Misclassification Cost
A missed fraud and a blocked legitimate payment have different consequences. Compare those costs when choosing the action rule.
-
Decision Threshold
The model produces a score. Someone decides which scores lead to a call, a block, or a review, and that someone is usually not the person who trained the model.
-
Subsampling
Asking sklearn.utils.resample for 2,000 rows returned 1,890 distinct ones. The default is replace=True, which is a bootstrap, not a subsample.
-
Class Weighting
Balanced class weights changed the Brier score from 0.01231 to 0.10726. A threshold change can match recall without reproducing the weighted model.
-
SMOTE (Synthetic Minority Over-sampling Technique)
SMOTE took the average predicted risk from 0.0240 to 0.2572 against a true rate of 0.0247, while AUROC moved by only 0.012.
-
Oversampling
Oversampling the minority four times gave coefficients identical to class_weight. The ranking barely moved; the average predicted risk more than doubled.
-
NearMiss (Distance-based Undersampling)
Average precision: no undersampling 0.6740, random 0.6586, NearMiss-1 0.6551, NearMiss-3 0.6358, NearMiss-2 0.1719. AUROC barely showed the damage.
-
Cluster-based undersampling
ClusterCentroids kept 344 majority rows, of which zero were real observations. They are k-means centroids — synthetic points, not selected samples.
-
Random Undersampling
Undersampling cut 14,000 training rows to 688. AUROC went up slightly; the average predicted risk went from 0.0240 to 0.2789 against a true rate of 0.0247.
-
Micro AUROC
Rescaling one class’s column leaves every per-class AUROC untouched and moves micro AUROC by 0.017. Here is why.
-
Multi-label Classification
Mean pairwise label correlation of +0.5035, and subset accuracy fell from 0.8515 to 0.4278 as the label count went from one to eight.
-
Micro F1 Score
The identity that makes micro precision, recall and F1 equal has one arithmetic reason — and an example that violates it cannot be real.
-
Single-label Classification
Forcing one label onto overlapping data left 57.38% of rows impossible to describe correctly, and one answer recovered 13.67% of the real tags.
-
Micro Recall
Per class the false positives and false negatives look nothing alike. Their totals are identical, and that is the whole mechanism.
-
Micro Precision
With one true and predicted label per sample and all classes included, micro precision, recall, and F1 equal accuracy. Label subsets and multilabel data break the identity.
-
One-vs-Rest (OvR) AUROC
Each OvR AUROC compares one class with its own negative mixture. Interpret the mixture and finite-sample variation before comparing class scores.
-
Macro AUROC (Macro-Averaged AUROC)
Macro AUROC weights classes equally, but OvR still depends on the negative-class mixture. OvO, uncertainty, and thresholds need separate interpretation.
-
Macro F1
Averaging then combining and combining then averaging give different answers. Which one your report contains is worth knowing.
-
Macro Recall
Macro recall and unadjusted balanced accuracy match on the same present classes. Absent labels and zero-division conventions can change that comparison.
-
Macro Precision
The same predictions can score 0.27, 0.81 or 0.94 macro precision. What changed was zero_division, not the model.
-
Multiclass AUROC
One-vs-rest or one-vs-one, macro or weighted or micro. Which combination produced the number matters more than the number.
-
Gini Coefficient
The same credit scores gave a credit-risk Gini of 0.6706 and an economics Gini of 0.4601. The analogy between the two curves is not an identity.
-
Predictive Parity (Calibration)
Predictive parity compares precision after a decision. Calibration concerns probabilities. Small examples show why neither condition guarantees the other.
-
Equalized Odds (Fairness)
Equalized odds compares true and false positive rates within groups. Learn its denominators, relation to other criteria, threshold choices and practical limits.
-
Equal Opportunity
Equal opportunity compares recall among positives. An example explains what it protects, which errors it leaves unconstrained and how labels affect it.
-
Demographic Parity (Statistical Parity)
Demographic parity compares selection shares. Learn how to calculate it, interpret thresholds and assess what equal rates leave unresolved.
-
Thompson Sampling
Thompson sampling draws a plausible success rate for each arm from its posterior and plays the largest draw. Uncertain arms get tried in proportion to how likely they are to be best.
-
Gaussian Processes (GPs)
Standard deviation of 0.1123 where observations were dense, 0.9620 in the gap between them, and 1.0046 beyond all the data.
-
Regression Coefficient
The same model gave income a coefficient of 3.99 or 3,994.80 depending on whether income was counted in dollars or thousands. R-squared was 0.7470 either way.
-
Ranking Algorithms
A ranking algorithm turns candidates into an ordered list. Pairwise learning to rank trains from item preferences; lambda methods use the effect of swaps on a metric such as NDCG to guide training.
-
Probabilistic Interleaving
Instead of copying items from two fixed lists, probabilistic interleaving samples each slot from distributions built from the rankings, so small ranking differences change the credit.
-
Team Draft Interleaving (TDI)
Each round a coin flip decides which ranker picks first. Every item belongs to the ranker that picked it, so each click has exactly one owner.
-
Balanced Interleaving
Balanced interleaving alternates between two rankings and counts clicks in each ranking’s top k. With two rankings that differ only by a shift, one ranker can win most random clicks.
-
Bandit Algorithms
A bandit algorithm chooses among options while it learns which is best. Regret measures what the learning cost compared with always choosing the best option.
-
Blocked Splits (Single Holdout)
Moving one cut point across a series changed the MAE from 1.19 to 3.18 — same data, same model, nine defensible splits.
-
Sliding Window (Rolling Window) Cross-Validation
After a regime change, the sliding window recovers to an MAE of 0.43 while the expanding window stalls at 2.79.
-
Expanding Window Validation
The first fold trained on 200 rows scores 1.77 and the last on 1,000 scores 0.87. Their mean summarizes this sequence of historical evaluations.
-
Data Leakage
Leakage can distort evaluation through future features, preprocessing, or test feedback. Time and group splits help only when they match the intended task.
-
Stratified Group K-Fold
A chance-level problem scored 0.9198 when groups were split across folds and 0.5323 when they were kept together. The truth was 0.5000.
-
Stratified Shuffle Split
Five splits of 1,000 rows tested 1,000 draws — but only 679 distinct rows. The other 321 were never in a test set at all.
-
Multiclass stratified CV
A class with three samples cannot appear in five folds. scikit-learn warns instead of failing, and the warning is easy to miss.
-
k-fold cross-validation
Selecting features on pure noise before the split turns a chance-level problem into 90 percent accuracy. Inside a pipeline it reads 0.53.
-
Cross-Validation (CV)
On one dataset, five-fold CV ranges from 0.47 to 0.62 across split seeds. A separate ten-fold simulation illustrates why naive fold error bars can be too small.
-
Early Stopping
Stopping where the validation set said cost 0.0010 of test loss against the best available. Not stopping at all cost 0.0965.