Binary Classification

Binary classification assigns each input to one of two classes — spam or not, churn or retain, disease present or absent. It is the base case that multiclass and multi-label problems are built out of.

Most models output a probability \(p \in [0, 1]\) for the positive class, and a threshold turns it into an answer:

\[ \hat{y} = \begin{cases} 1 & \text{if } p \geq t \\ 0 & \text{if } p < t \end{cases} \]

Textbooks write \(t = 0.5\) and move on. That number is a convention, not a property of the model, and it is where most of the practical difficulty lives.

The threshold is the decision

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=20000, n_features=8, n_informative=5,
                           weights=[0.92], flip_y=0.01, random_state=0)
xtr, xte, ytr, yte = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0)
p = LogisticRegression(max_iter=5000).fit(xtr, ytr).predict_proba(xte)[:, 1]
"""One model, one set of predicted probabilities. Only the cut-off changes below."""

print(f"positives in the test set {yte.mean():.4f}   AUROC {roc_auc_score(yte, p):.4f} (unchanged by any threshold)\n")
print(f"{'threshold':>10}  {'accuracy':>9} {'precision':>10} {'recall':>8} {'F1':>7}  flagged")
for t in (0.10, 0.25, 0.50, 0.75, 0.90):
    hit = p >= t
    print(f"{t:>10.2f}  {accuracy_score(yte, hit):>9.4f} {precision_score(yte, hit, zero_division=0):>10.4f}"
          f" {recall_score(yte, hit):>8.4f} {f1_score(yte, hit):>7.4f}  {hit.mean():>7.4f}")
print(f"{'flag none':>10}  {accuracy_score(yte, np.zeros_like(yte)):>9.4f} {0.0:>10.4f} {0.0:>8.4f} {0.0:>7.4f}  {0.0:>7.4f}")
# positives in the test set 0.0840   AUROC 0.9524 (unchanged by any threshold)
#
#  threshold   accuracy  precision   recall      F1  flagged
#       0.10     0.9237     0.5271   0.8869  0.6612   0.1413
#       0.25     0.9630     0.7670   0.8036  0.7849   0.0880
#       0.50     0.9687     0.8930   0.7123  0.7925   0.0670
#       0.75     0.9652     0.9773   0.5992  0.7429   0.0515
#       0.90     0.9517     0.9864   0.4306  0.5994   0.0367
#  flag none     0.9160     0.0000   0.0000  0.0000   0.0000

One model. One set of predicted probabilities. Nothing is refitted between rows of that table.

Precision runs from 0.5271 to 0.9864 and recall from 0.8869 to 0.4306, purely as a function of where the cut-off goes. AUROC sits at 0.9524 throughout, because it summarizes the ranking rather than any one cut-off.

So two teams deploying the same model can report completely different performance and both be right. What separates them is a choice nobody wrote down.

For calibrated probabilities and constant false-positive and false-negative costs, with zero cost for correct decisions, the cost-minimizing threshold is set by those costs. Where a miss costs ten times a false alarm, the cut-off belongs well below 0.5. Where those costs are unknown — which is usual — the binding constraint is normally operational: how many flags can be reviewed per day. Either way the number comes from outside the model, and setting it deliberately is part of the work.

Accuracy is the wrong summary

The last line of the run is the one to remember. Positives are 8.40% of this test set, so a rule that flags nothing at all scores 0.9160 accuracy. The best threshold in the table reaches 0.9687. The entire value of the model, measured in accuracy, is 5.3 points above doing nothing.

This is why the metric list matters. Precision, recall and their harmonic mean describe what happens to the positives; accuracy is dominated by the negatives and hides the part you care about. On rare-event problems average precision is usually the honest single number, and AUROC can stay high while the operating point degrades badly.

The other summary worth having is calibration: whether \(p = 0.30\) actually means three times in ten. A model can rank perfectly and still be wrong about the level, which matters as soon as the probability feeds a cost calculation rather than a cut-off.

Reporting one

• State the threshold and why it is there. Without it, precision and recall are not reproducible.
• Give the base rate beside accuracy, so a reader can compute the do-nothing baseline.
• Report a threshold-free number too — AUROC or average precision — so that a later threshold change can be separated from a real change in the model.
• Say which class is positive. Precision and recall are not symmetric, and the convention is not self-evident from a confusion matrix.

With 1,000 emails, 200 of them spam, a model that flags 220 and gets 180 of them right has precision 180/220 = 0.818, recall 180/200 = 0.90 and F1 0.857. That is a complete report — and it is complete because the counts are there, not because the three ratios are.

References

• scikit-learn User Guide, Metrics and scoring