Accuracy

Accuracy is the share of predictions a model gets right:

\(\text{Accuracy} = \dfrac{TP + TN}{TP + TN + FP + FN}\)

where \(TP\) and \(TN\) are the correctly classified positives and negatives, and \(FP\) and \(FN\) are the two kinds of mistake. A spam filter that catches 90 of 100 spam messages while wrongly flagging 20 of 900 legitimate ones gets \(90 + 880 = 970\) of 1,000 right, so its accuracy is 97 percent.

It is the first number anyone computes and the easiest to explain. It is also the one most likely to be quoted without the context that gives it meaning.

The number it has to be compared against

A model that always predicts the majority class gets \(\max(\pi, 1-\pi)\) right, where \(\pi\) is the proportion of positives. That rule learns nothing and reads no features, and it sets the floor any accuracy figure has to clear.

import numpy as np

def scores(tp, fp, fn, tn):
    total = tp + fp + fn + tn
    precision = tp / (tp + fp) if tp + fp else float("nan")
    recall = tp / (tp + fn) if tp + fn else float("nan")
    return (tp + tn) / total, precision, recall

N, PREVALENCE = 10000, 0.10
positives = int(N * PREVALENCE)
negatives = N - positives

cases = {
    "always negative": (0, 0, positives, negatives),
    "misses half":     (500, 0, 500, negatives),
    "over-flags":      (positives, 500, 0, negatives - 500),
}
for name, (tp, fp, fn, tn) in cases.items():
    acc, precision, recall = scores(tp, fp, fn, tn)
    print(f"{name:16} accuracy {acc:.4f}  precision {precision:.4f}  recall {recall:.4f}")

print()
for prevalence in (0.50, 0.10, 0.01, 0.001):
    print(f"prevalence {prevalence:<6} always-negative accuracy {1 - prevalence:.4f}")
# always negative  accuracy 0.9000  precision nan  recall 0.0000
# misses half      accuracy 0.9500  precision 1.0000  recall 0.5000
# over-flags       accuracy 0.9500  precision 0.6667  recall 1.0000
#
# prevalence 0.5    always-negative accuracy 0.5000
# prevalence 0.1    always-negative accuracy 0.9000
# prevalence 0.01   always-negative accuracy 0.9900
# prevalence 0.001  always-negative accuracy 0.9990

The bottom half of that output is the whole problem with quoting accuracy alone. At a prevalence of 0.1 percent — rare disease, rare fraud, rare defect — the do-nothing rule scores 99.9 percent. A model reported at 99.5 percent accuracy in that setting has lower accuracy than the always-negative baseline. Its practical value still depends on which errors it makes and their costs.

The top half makes a second point. Both example prediction profiles score 0.9500, and they are not remotely the same model. One flags nobody it should not and misses half the positives; the other catches every positive and raises 500 false alarms. Accuracy cannot tell them apart because it adds \(FP\) and \(FN\) together, and whether that is acceptable depends entirely on which error costs more.

What the equal weighting assumes

Accuracy is usually described as safe when the classes are balanced. That is not quite the condition. Adding \(FP\) and \(FN\) assumes they cost the same — a 0/1 cost matrix — and class balance has nothing to do with whether that is true. Missing a tumor and calling a healthy patient back for a second scan are both one error to accuracy, on a perfectly balanced dataset.

The same assumption is hidden in the cutoff. Accuracy is computed after a threshold has been applied, and a cutoff of 0.5 minimizes expected 0/1 error when the score is the positive-class posterior for the target population and the two errors have equal cost. Elkan noted that standard learning algorithms “implicitly make decisions based on the probability threshold 0.5”, so an accuracy figure inherits that choice without stating it. Where the costs differ, the threshold should move — see recalibrate thresholds — and the accuracy reported at the old cutoff describes a system nobody would deploy.

This is why accuracy fell out of favor as a primary metric. Fawcett attributes the rise of ROC analysis in machine learning partly to “the realization that simple classification accuracy is often a poor metric for measuring performance”.

When it is the right number

• Equal per-case error costs and representative evaluation cases. Balanced classes make minority performance less easily hidden, but class balance alone does not establish that accuracy matches the task.
• As a sanity check during development. Accuracy below the majority baseline is a reason to check labels, thresholds, sampling and model fit; it does not by itself identify the cause.
• When the audience needs one number and will not read a second. Even then, report the baseline alongside it.

What to report instead

Balanced accuracy deserves a note: it replaces the pooled count with the mean of per-class recall, so the do-nothing rule scores 0.5 regardless of prevalence. That removes the baseline problem, and it replaces rather than removes the cost assumption: weighting the classes equally is itself a choice, equivalent to costs that scale inversely with class size rather than to equal per-case costs.

References

• Tom Fawcett, “An introduction to ROC analysis”, Pattern Recognition Letters, 27(8), 2006, 861–874
• Charles Elkan, “The Foundations of Cost-Sensitive Learning”, Proceedings of IJCAI 2001