Performance Management and Talent Development
The engineer who kept settlement running got a middling rating and left with no successor. Results, behaviors, levels, calibration, hiring, and who is next.
-
-
Class Imbalance
When fraud is one case in a thousand, a model that never flags fraud is 99.9% accurate. Imbalance is a fact to evaluate around, not a defect to remove.
-
Decision Threshold
The model produces a score. Someone decides which scores lead to a call, a block, or a review, and that someone is usually not the person who trained the model.
-
Class Weighting
Balanced class weights changed the Brier score from 0.01231 to 0.10726. A threshold change can match recall without reproducing the weighted model.
-
SMOTE (Synthetic Minority Over-sampling Technique)
SMOTE took the average predicted risk from 0.0240 to 0.2572 against a true rate of 0.0247, while AUROC moved by only 0.012.
-
Oversampling
Oversampling the minority four times gave coefficients identical to class_weight. The ranking barely moved; the average predicted risk more than doubled.
-
Random Undersampling
Undersampling cut 14,000 training rows to 688. AUROC went up slightly; the average predicted risk went from 0.0240 to 0.2789 against a true rate of 0.0247.
-
Micro AUROC
Rescaling one class’s column leaves every per-class AUROC untouched and moves micro AUROC by 0.017. Here is why.
-
Predictive Parity (Calibration)
Predictive parity compares precision after a decision. Calibration concerns probabilities. Small examples show why neither condition guarantees the other.
-
Predictive LTV (pLTV)
Forecast customer value with a clear cutoff and horizon, choose a model for the purchasing process, and validate the decision it supports.
-
Posterior belief
Posterior beliefs depend on stated assumptions. Coherence, interval coverage, prior sensitivity, and new evidence test different aspects of the claim.
-
Re-scoring
The same +0.2 boost moves promoted items in one score band up 989 places on average and items in another band 366, and it does least at the top of the page.
-
Recalibration
Platt, isotonic, temperature and beta calibration — how to choose, and the measured cost of recalibrating a model that was already fine.
-
Bayesian Correction
A source probability of 0.80 at a balanced prior becomes 0.3077 at a target prior of 0.10, provided the label-shift and probability assumptions hold.
-
Recalibrate Thresholds
Learn how to derive a decision cutoff, tune it without leakage, and reassess it after changes to probabilities, costs, data or alert behavior.
-
Lagging Indicators
AUC uncertainty depends on positive and negative counts and paired-score dependence, not just the AUC values.
-
Classification Probability
What models output per class, how they produce it, and the measured gap between a number in [0, 1] and a number you can act on.
-
Probability Forecasts
Days it calls 0.0152 happen 23.81% of the time, and accuracy ranks it level with the truth.
-
Adaptive ECE
Equal-count bins instead of equal-width ones. What that fixes, what it does not, and why it can report a larger error for the same model.
-
Maximum Calibration Error (MCE)
The largest single deviation between predicted probability and observed frequency. Useful for spotting a dangerous region, fragile as a headline number.
-
Murphy’s Decomposition
Use reliability, resolution and uncertainty to interpret a Brier score. A small exact example explains the terms and the limits of binned estimates.
-
Temperature Scaling
A single parameter that softens or sharpens a network’s probabilities without touching which class it picks — and what that single parameter cannot fix.
-
Platt Scaling
A two-parameter logistic map from scores to probabilities, fitted on held-out data. An increasing logistic map on one scalar score preserves its ranking; a negative fitted slope can reverse it.
-
Isotonic Regression
Non-decreasing fitted values can correct monotone probability distortions. Scikit-learn uses linear interpolation between fitted knots, and small calibration samples can overfit.
-
Support Vector Machines (SVMs)
predict() and predict_proba() >= 0.5 disagreed on 25 of 3,000 training rows, and the model kept 48.2% of them as support vectors.
-
Underconfident
Probabilities too close to 0.5, intervals wider than the evidence requires. The mirror of overconfidence, and repairable the same way.
-
Overconfident
Probabilities too close to 0 and 1, intervals too narrow. A monotone sharpening can preserve accuracy at 0.5 and AUC while worsening probability estimates.
-
Binary Cross-Entropy (BCE)
On events that happen 30% of the time, reporting 0.30 gave the lowest average loss at 0.610682. Reporting 0.20 or 0.40 both cost more.
-
Logit Space
Why models work on the log-odds scale, and how an additive constant there corresponds to the same nonlinear correction on the probability scale.
-
Logistic Regression
A coefficient of +0.8022 multiplies the odds by 2.2305 everywhere — but raises the probability by 0.0849, 0.1903 or 0.0437 depending where you stand.
-
Softmax Function
The same logits read [0.9817, 0.0180, 0.0003] at temperature 0.25 and [0.3672, 0.3322, 0.3006] at temperature 10.
-
One-vs-Rest (OvR)
The five binary models gave probabilities summing to 1.0452, 0.9266, 1.0837 and 1.0159 on four test rows. predict_proba divides by the sum.
-
Deep Ensembles
Deep ensembles average separately trained networks. Error correlation limits gains, and member disagreement captures only part of predictive uncertainty.
-
Full Annotation
Leaving 80% of training positives unlabeled reduced recall from 0.6341 to 0.0031 in this experiment. Evaluation-label omissions have different effects.
-
Fairness parity
Equal error rates can coexist with unequal selection rates and precision. A worked example explains the denominators, trade-offs and limits of parity claims.
-
Label Drift
A changed class mix can alter calibrated probabilities under label-shift assumptions. Check those assumptions before choosing recalibration or retraining.
-
Expected Calibration Error (ECE)
One number for how far predicted probabilities sit from observed frequencies — biased upward, binning-dependent, and defined two different ways.
-
Reliability Curves
The diagnostic that shows where a model’s probabilities go wrong rather than how much. Including how much wobble to expect from a model that is right.
-
Brier Score
Mean squared error for probabilities. It rewards calibration and discrimination at once, which is why it needs a baseline before it means anything.
-
Calibration quality (Model Calibration)
Learn what a calibrated probability means, how calibration differs from ranking, and how to diagnose and improve it without confusing scores with decisions.
-
Uncertainty Estimation and Model Calibration
A softmax output of 0.99 is not a 99% chance of being right. Here is how far apart those two things are.
-
Fairness and Bias Auditing
Fairness criteria can conflict under stated conditions. An audit must define the decision, population, and criterion being compared.
-
Probability Calibration
Ranking well and reporting honest probabilities are different skills.
-
Loss Functions and Convex Surrogates
The loss function decides what a model estimates and how it treats large errors; decisions and costs come after.