Sensitivity and Specificity

Sensitivity and specificity describe a binary classifier by how it behaves on each true class separately. Sensitivity is the share of actual positives that it labels positive. Specificity is the share of actual negatives that it labels negative. Because each is computed within one class, neither depends on how common that class is.

Definitions

From the confusion matrix with true positives TP, false negatives FN, false positives FP and true negatives TN:

\[ \text{sensitivity} = \frac{TP}{TP+FN}, \qquad \text{specificity} = \frac{TN}{TN+FP}. \]

Sensitivity is the same quantity as recall and as the true positive rate. Specificity is the true negative rate, which equals one minus the false positive rate.

Worked example (hypothetical)

A judge reviews 1,000 claims, 50 of which are truly unsupported ("positive" here means the failure class). It flags 40 of those 50 and also flags 57 of the 950 supported claims.

• Sensitivity: 40/50 = 0.80.
• Specificity: 893/950 = 0.94.
• Accuracy: (40 + 893)/1000 = 0.933.
• Precision, the share of flags that are real: 40/97 = 0.41.

A judge that passes everything would have accuracy 0.95, higher than this judge, with sensitivity 0 and specificity 1. Accuracy rewards ignoring the rare class, so it is a poor way to validate a judge when failures are rare. Reporting both rates, per failure type, exposes this.

Why precision is different

Precision mixes the two rates with the prevalence of the positive class. With sensitivity 0.80 and specificity 0.94:

Prevalence of failures	Precision
50%	0.93
20%	0.77
5%	0.41
1%	0.12

The same judge looks reliable on a balanced test set and unhelpful on production traffic where failures are rare, since the false flags come from the large pool of good cases. This is why sensitivity and specificity are used to describe a measuring instrument, and precision to describe what a given flag is worth in a particular population.

Use in judge validation

1. Collect an audit sample labeled by people, with enough true failures to estimate sensitivity; oversample the rare class if necessary.
2. Compute both rates, with intervals (the Wilson interval is a standard choice for a proportion), per failure type and per stratum.
3. Use the rates to correct an aggregate measurement; see the Rogan–Gladen correction.

The pair is related to Youden's J = sensitivity + specificity − 1, which is 0.74 here. A value of 0 means the judge's flags carry no information about the true class.

Trade-off and thresholds

When a classifier outputs a score, moving the decision threshold trades one rate for the other: a lower threshold flags more, raising sensitivity and lowering specificity. The ROC curve plots sensitivity against 1 − specificity over all thresholds. For a fixed system, report the rates at the threshold actually used.