Hyperparameter
Trying 300 configurations instead of one gained 0.0267 on the validation set and only 0.0082 in reality — and past 30, the real score fell.
-
-
AI (Artificial Intelligence)
AI includes reasoning, search, learning, and decision-making. Rules, learned models, generative capabilities, and acting systems play different roles.
-
Machine Learning (ML)
One model scored a perfect 1.0000 on the rows it was trained on and 0.3474 on new ones. Another scored 0.9842 and 0.9822.
-
Ensemble
Learn how ensemble methods combine models, what bias and variance do and do not explain, and how to avoid leakage when learning a combination.
-
Recalibration
Platt, isotonic, temperature and beta calibration — how to choose, and the measured cost of recalibrating a model that was already fine.
-
Active Learning
From a 10-label seed, uncertainty sampling reached 0.5714 and random sampling 0.6418 on the same budget. It only won once the seed reached 400 in this experiment.
-
Bayesian Correction
A source probability of 0.80 at a balanced prior becomes 0.3077 at a target prior of 0.10, provided the label-shift and probability assumptions hold.
-
Recalibrate Thresholds
Learn how to derive a decision cutoff, tune it without leakage, and reassess it after changes to probabilities, costs, data or alert behavior.
-
Cardinality in Categorical Data
A distinct count alone cannot choose an encoding. Learn how cardinality interacts with category frequency, statistical evidence and unseen values.
-
Categorical Explosions
High-cardinality features create storage, estimation and unseen-category problems. Learn which encoding choices address each problem and what they sacrifice.
-
Discriminatory Power
AUROC and Gini preferred one model, KS and average precision preferred the other. All four are described as measures of discriminatory power.
-
Feature Values
A feature value is what one feature holds for one example. Models need numbers, so categories are encoded, numeric ranges are scaled, and missing values need an explicit rule.
-
Four-Fifths Rule
With twenty applicants per group and no difference at all in the process, the four-fifths rule indicated adverse impact in 61.7% of simulated runs.
-
Classification Probability
What models output per class, how they produce it, and the measured gap between a number in [0, 1] and a number you can act on.
-
Strictly Proper Scoring Rules
The property that makes a probability metric ungameable: honest reporting is uniquely optimal. Shown by calculation, including a rule that lacks it.
-
Target Variable
Defining churn as leaving within 7 days versus 365 moved the positive rate from 0.2318 to 0.9879 and AUROC from 0.7452 to 0.9619 on identical customers.
-
Simple Baseline Methods
On a trend the mean method scored MAE 15.8737 against drift’s 2.2367. On a seasonal series, seasonal naive won at 1.7016 against naive’s 3.0688.
-
Long-Tail Items
Interaction data are usually dominated by a few popular items. Long-tail items are the many items with few interactions, and there is no universal cutoff for where the tail begins.
-
Self-Information of Popularity
Self-information is the surprise of an event, −log₂ of its probability. Applied to popularity, an item few users have chosen scores high, and the average over a list measures novelty.
-
Relevance in Recommender Systems
A ranking metric can only reward what the labels call relevant. Judged, rated, clicked, and held-out labels measure different things, and a missing label is not a negative label.
-
Genre Overlap
Genre overlap is not a single standard metric. It is a set-overlap measure applied to genre tags, and the choice of formula changes the value.
-
Jaccard index
The Jaccard index is the size of the intersection divided by the size of the union. It ignores weights, and two empty sets need an explicit rule.
-
Intra-List Diversity (ILD)
ILD is the average distance between the items in one recommendation list. The distance function and the list length decide what the number means.
-
Dominating in Recommender Systems
One model dominates another when it is at least as good on every metric and better on at least one. Most comparisons end with several non-dominated models and a trade-off to decide.
-
User Coverage
User coverage is the share of users who actually receive recommendations. A system that withholds low-confidence recommendations can gain accuracy and lose coverage.
-
Item Coverage
Item coverage counts how much of the catalog the system ever recommends. It grows with the number of users and the list length, and it can be high while a few items dominate.
-
Diminishing Utility
Understand total versus marginal utility, why concavity produces risk aversion under expected utility, and how diminishing returns differ in machine learning.
-
DCG (Discounted Cumulative Gain)
DCG adds up the relevance of ranked items but discounts lower positions. Linear and exponential gains give different numbers, so the convention has to travel with the score.
-
Kaggle
Kaggle hosts machine learning competitions ranked on held-out data. A large study found little evidence of substantial leaderboard overfitting, but a competition score still measures one fixed task.
-
TREC (Text REtrieval Conference)
TREC gives researchers shared documents, questions, and relevance judgments so systems can be compared on one task. Its pooling method decides which documents get judged.
-
Adaptive ECE
Equal-count bins instead of equal-width ones. What that fixes, what it does not, and why it can report a larger error for the same model.
-
Maximum Calibration Error (MCE)
The largest single deviation between predicted probability and observed frequency. Useful for spotting a dangerous region, fragile as a headline number.
-
ROC Curve (Receiver Operating Characteristic)
What the curve shows, why it is invariant to prevalence-only shifts and strictly increasing rescaling, and where that invariance becomes a blind spot.
-
Murphy’s Decomposition
Use reliability, resolution and uncertainty to interpret a Brier score. A small exact example explains the terms and the limits of binned estimates.
-
Temperature Scaling
A single parameter that softens or sharpens a network’s probabilities without touching which class it picks — and what that single parameter cannot fix.
-
Platt Scaling
A two-parameter logistic map from scores to probabilities, fitted on held-out data. An increasing logistic map on one scalar score preserves its ranking; a negative fitted slope can reverse it.
-
Isotonic Regression
Non-decreasing fitted values can correct monotone probability distortions. Scikit-learn uses linear interpolation between fitted knots, and small calibration samples can overfit.
-
Support Vector Machines (SVMs)
predict() and predict_proba() >= 0.5 disagreed on 25 of 3,000 training rows, and the model kept 48.2% of them as support vectors.
-
Underconfident
Probabilities too close to 0.5, intervals wider than the evidence requires. The mirror of overconfidence, and repairable the same way.
-
Overconfident
Probabilities too close to 0 and 1, intervals too narrow. A monotone sharpening can preserve accuracy at 0.5 and AUC while worsening probability estimates.
-
Binary Cross-Entropy (BCE)
On events that happen 30% of the time, reporting 0.30 gave the lowest average loss at 0.610682. Reporting 0.20 or 0.40 both cost more.
-
Logit Space
Why models work on the log-odds scale, and how an additive constant there corresponds to the same nonlinear correction on the probability scale.
-
Logistic Regression
A coefficient of +0.8022 multiplies the odds by 2.2305 everywhere — but raises the probability by 0.0849, 0.1903 or 0.0437 depending where you stand.
-
Binary Classification
One model, one set of probabilities. Moving the cut-off from 0.10 to 0.90 took precision from 0.5271 to 0.9864 and recall from 0.8869 to 0.4306.
-
Classification Models
Training on the number and cutting afterward matched the classifier’s AUROC of 0.9967 — and also answered the numeric question at R-squared 0.9900.
-
Log-Odds
Odds, log-odds, and why the same coefficient moves a probability by 0.01 in one place and 0.17 in another.
-
Outlier
With one contaminating value the largest z-score was 17.8407. With thirty of them it was 4.0420 — they inflate the deviation they are measured against.
-
Regression Models
A regression predicts an outcome; it does not tell you what would happen if you changed an input. Those are different questions with different requirements.
-
One-vs-Rest (OvR)
The five binary models gave probabilities summing to 1.0452, 0.9266, 1.0837 and 1.0159 on four test rows. predict_proba divides by the sum.
-
Multiclass Classification
Accuracy 0.8575 and micro F1 0.8575 — and the rarest class had a recall of 0.0382. Macro F1 came out at 0.6922.