Binary Classification Models – Conceptual Framework and Evaluation Metrics
The State of a Binary Classification Model
A binary classification model is a special case of a nominal classification model in which the label variable has exactly two categories.
Let the label variable be denoted by $y$.
- The two label categories are $c_1$ and $c_2$, so the number of categories is $k = 2$.
- The observed labels in the dataset are $y_i$, for $i = 1, \dots, n$.
- The predicted class labels produced by the model are $\hat y_i$, for $i = 1, \dots, n$.
- The model also produces predicted probabilities $\hat p_{ij}$, where:
- $j \in {1,2}$,
- $\hat p_{ij}$ is the probability that observation $i$ belongs to category $c_j$,
- $\hat p_{i1} + \hat p_{i2} = 1$ for every observation.
Thus, each observation is associated with both a predicted class and a probability distribution over the two classes.
Event and Non-Event Categories
Without loss of generality, the two label categories are assigned distinct roles:
- $c_1$ is defined as the Event category, meaning it is the outcome of primary interest.
- $c_2$ is defined as the Non-Event category.
Accordingly:
- $\hat p_{i1}$ is called the predicted Event probability,
- $\hat p_{i2}$ is called the predicted Non-Event probability,
- $\hat p_{i1} + \hat p_{i2} = 1$ always holds.
Binary classification performance is therefore driven primarily by how well $\hat p_{i1}$ separates Events from Non-Events.
Classification via a Probability Threshold
To convert predicted probabilities into predicted class labels, a threshold $t$ is introduced, where $0 \le t \le 1$.
The classification rule is defined as follows:
- If $\hat p_{i1} \ge t$, then $\hat y_i = c_1$ (Event).
- If $\hat p_{i1} < t$, then $\hat y_i = c_2$ (Non-Event).
Special cases highlight the role of the threshold:
- If $t = 0$, all observations are classified as Events.
- If $t = 1$, all observations are classified as Non-Events, except those with $\hat p_{i1} = 1$.
Thus, threshold selection directly controls the balance between Event and Non-Event predictions.
Strategies for Choosing the Threshold
Selecting an appropriate threshold is a modeling decision, not a mathematical necessity. Common strategies include:
- Empirical choice
The threshold is set equal to the observed proportion of Events in the training data or in the full dataset. - Uninformative choice
The threshold is set to $t = 0.5$, assigning equal prior importance to both classes. - Model-directed choice
The threshold is chosen to optimize a specific model metric, such as misclassification rate, sensitivity, or profit-based criteria.
The Confusion Matrix
Once a threshold is fixed, classification outcomes can be summarized using the confusion matrix, which compares observed labels with predicted labels.
| Observed \ Predicted | Event | Non-Event |
|---|---|---|
| Event | True Positive (TP) | False Negative (FN) |
| Non-Event | False Positive (FP) | True Negative (TN) |
From the confusion matrix, several important quantities are defined:
- Sensitivity (True Positive Rate):
$\text{Sensitivity} = \frac{\text{TP}}{\text{TP} + \text{FN}}$ - Specificity (True Negative Rate):
$\text{Specificity} = \frac{\text{TN}}{\text{FP} + \text{TN}}$ - Misclassification Rate:
$\frac{\text{FP} + \text{FN}}{\text{TP} + \text{FP} + \text{FN} + \text{TN}}$
The misclassification rate measures the proportion of incorrect classifications across all observations.
Misclassification Rate and RASE in Binary Classification
A binary classification model is a special case of a nominal classification model with $k = 2$. As a result, both the Misclassification Rate and the Root Average Squared Error (RASE) defined for nominal classification apply directly.
For binary classification:
- The RASE value is bounded between 0 and 1.
- A useful model should have $\text{RASE} < 0.5$.
- Smaller RASE values indicate better probabilistic predictions.
Simplification of RASE When $k = 2$
In nominal classification, RASE is defined as:
$\text{RASE} = \sqrt{\frac{1}{2n} \sum_{i=1}^n \sum_{j=1}^2 (\delta_{ij} – \hat p_{ij})^2}$
When $k = 2$, the indicator variables satisfy $\delta_{i1} + \delta_{i2} = 1$, and the predicted probabilities satisfy $\hat p_{i1} + \hat p_{i2} = 1$.
Using these identities:
$(\delta_{i1} – \hat p_{i1})^2 + (\delta_{i2} – \hat p_{i2})^2 = 2(\delta_{i1} – \hat p_{i1})^2$
Therefore, RASE simplifies to:
$\text{RASE} = \sqrt{\frac{1}{n} \sum_{i=1}^n (\delta_{i1} – \hat p_{i1})^2}$
This shows that only the predicted Event probability $\hat p_{i1}$ is required to compute RASE in binary classification.
A Probability-Ranking Perspective
Binary classification quality can also be assessed through pairwise probability comparisons.
The predicted Event probabilities $\hat p_{i1}$ are divided into two groups:
- $\hat p_{i1}^E$: probabilities associated with observations where $y_i = c_1$ (Event),
- $\hat p_{i1}^{NE}$: probabilities associated with observations where $y_i = c_2$ (Non-Event).
Let $n_E$ denote the number of Event observations and $n_{NE}$ denote the number of Non-Event observations.
There are $n_E \times n_{NE}$ possible pairs consisting of one Event and one Non-Event observation.
Concordant, Discordant, and Tied Pairs
For a randomly selected pair $(r, s)$ such that $y_r = c_1$ and $y_s = c_2$:
- The pair is Concordant if $\hat p_{r1}^E > \hat p_{s1}^{NE}$.
- The pair is Discordant if $\hat p_{r1}^E < \hat p_{s1}^{NE}$.
- The pair is Tied if $\hat p_{r1}^E = \hat p_{s1}^{NE}$.
Let:
- $C$ be the number of concordant pairs,
- $D$ be the number of discordant pairs,
- $T$ be the number of tied pairs.
By construction, $C + D + T = n_E \times n_{NE}$.
Area Under the Curve (AUC)
The Area Under the Curve (AUC) is defined as:
$\text{AUC} = \frac{1}{2} + \frac{C – D}{2(C + D + T)}$
AUC represents the probability that a randomly chosen Event observation receives a higher predicted Event probability than a randomly chosen Non-Event observation.
Key properties:
- If all pairs are concordant, AUC = 1.
- If all pairs are discordant, AUC = 0.
- If all pairs are tied, AUC = 0.5.
- If $C = D$ and there are no ties, AUC = 0.5.
A useful binary classification model should have AUC > 0.5, and higher AUC values indicate stronger discriminatory power.
Interpretation of AUC
AUC answers a probabilistic question:
What is the chance that the model assigns a higher Event probability to an Event observation than to a Non-Event observation?
Thus, AUC evaluates ranking quality, not classification accuracy at a specific threshold.
Logistic Regression Example Interpretation
In the logistic regression model presented:
- AUC = 0.7377, which is greater than 0.5.
- RASE = 0.4319, which is less than 0.5.
- Misclassification Rate = 0.2864, computed using the uninformative threshold $t = 0.5$.
These results indicate:
- The model separates Events from Non-Events better than chance.
- The predicted probabilities are meaningfully informative.
- There is room for improvement, but the model is acceptable by standard evaluation criteria.
Summary of Binary Classification Metrics
Binary classification models are evaluated using three complementary perspectives:
- Decision accuracy
Measured by misclassification rate and accuracy. - Probability calibration
Measured by RASE. - Ranking ability
Measured by AUC.
Together, these metrics provide a complete and rigorous assessment of binary classification performance.