Model Score

Model score means two unrelated things, and which one is meant is almost never stated.

In the first sense it is a per-case output: the raw value a model produces for one example before any cutoff is applied. In the second it is a per-model summary: the single number an evaluation returns for the whole test set. A paper reporting that positives receive a high model score means the first. A notebook line reading score = clf.score(X_test, y_test) means the second.

The raw output

Most classifiers compute something unbounded first and convert it to a probability afterward.

A worked example, using logistic regression on three transactions:

The raw score has no fixed scale. A +2.5 from one model and a +2.5 from another are not comparable, and neither is a +2.5 from the same model retrained on different data. What survives is the ordering, which is why AUC and average precision are computed from it directly and give the same answer whether the raw score or the probability is passed in.

The evaluation number

The usual scikit-learn supervised classifier and regressor mixins provide a score method: accuracy for classification and \(R^2\) for regression. Other estimator types, including some transformers, need not provide that method. It takes no metric argument, so the choice is invisible at the call site — which makes the second sense of model score the one most likely to be reported without anyone checking what it measured.

The two senses can point in opposite directions on the same fitted model.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(5)
n = 40000
x = rng.normal(size=(n, 1))
intercept = -3.2
"""The intercept makes positives rare, which is where the two senses of score diverge."""
p = 1 / (1 + np.exp(-(0.8 * x[:, 0] + intercept)))
y = (rng.uniform(size=n) < p).astype(int)
fit, test = slice(0, 20000), slice(20000, None)

clf = LogisticRegression().fit(x[fit], y[fit])
margin = clf.decision_function(x[test])
proba = clf.predict_proba(x[test])[:, 1]
"""margin is the raw score; proba is the same ranking squashed into [0, 1]."""

print(f"base rate      {y[test].mean():.4f}")
print(f"clf.score()    {clf.score(x[test], y[test]):.4f}")
print(f"always-0 rule  {1 - y[test].mean():.4f}")
print(f"AUC on margin  {roc_auc_score(y[test], margin):.4f}")
print(f"AUC on proba   {roc_auc_score(y[test], proba):.4f}")
print(f"margin range   {margin.min():.2f} {margin.max():.2f}")
# base rate      0.0522
# clf.score()    0.9478
# always-0 rule  0.9478
# AUC on margin  0.7147
# AUC on proba   0.7147
# margin range   -6.24 0.39

Positives make up 5.22 percent of the test set. The classifier’s score is 0.9478, exactly what labeling everything negative achieves. Not because the model predicts nothing: at a 0.5 cutoff it flags two cases out of 20,000, one right and one wrong, and the two cancel. The accuracy is the same to four decimal places and the model has contributed one correct alert and one false one. Read as a headline number it says the model is excellent. Read as a comparison against the trivial rule it says the model contributes nothing.

Yet the raw scores rank real positives above real negatives 71.47 percent of the time. The model has learned something; accuracy at this cutoff cannot see it, because almost every raw score falls below zero. The last line shows the range, -6.24 to 0.39: only two cases have a positive margin, so only two reach a probability of 0.5. The ranking is informative over the whole range and the cutoff is reading one end of it.

Two fixes apply to that situation and they are different. Moving the cutoff to where the costs put it is covered under recalibrate thresholds. Giving the raw score a meaningful scale so the cutoff can be computed rather than searched is recalibration.

Reading it in context

A raw score is not a probability, whatever its magnitude suggests. What it takes to turn it into one, and how to check whether it worked, is under classification probability.

References

• scikit-learn API reference, LogisticRegression
• scikit-learn API reference, LinearRegression
• Charles Elkan, “The Foundations of Cost-Sensitive Learning”, Proceedings of IJCAI 2001