Nominal Classification Models: Model State and Evaluation Metrics

The State of a Nominal Classification Model

Consider a nominal (categorical) classification problem, where the response variable has no inherent ordering.

• Let the label variable be denoted by $y$.
• The label can take one of $k$ distinct categories, written as $c_1, c_2, \dots, c_k$.
• The observed labels in the dataset are $y_i$, for $i = 1, \dots, n$.
• The model produces predicted class labels $\hat y_i$, for $i = 1, \dots, n$.
• In addition, the model outputs predicted probabilities $\hat p_{ij}$, where:
  • $\hat p_{ij}$ is the predicted probability that observation $i$ belongs to class $c_j$,
  • $j = 1, \dots, k$ and $i = 1, \dots, n$,
  • for each observation $i$, the probabilities satisfy
    $\sum_{j=1}^k \hat p_{ij} = 1$.

Thus, a nominal classification model produces both a discrete decision (the predicted class) and a probability distribution over all classes for each observation.

---

Misclassification Rate and Accuracy

A classification outcome is considered correct if the predicted class matches the observed class:

$y_i = \hat y_i$

A misclassification occurs when:

$y_i \ne \hat y_i$

Let $\#(y_i \ne \hat y_i)$ denote the total number of misclassified observations.
The Misclassification Rate is defined as:

$\text{Misclassification Rate} = \frac{\#(y_i \ne \hat y_i)}{n}$

Key properties:

• It takes values between 0 and 1 (inclusive).
• Smaller values indicate better classification performance.

The Accuracy is defined as:

$\text{Accuracy} = 1 – \text{Misclassification Rate}$

Accuracy represents the proportion of correctly classified observations and is the most commonly reported metric in classification tasks.

---

Classification Error at the Probability Level

While misclassification rate evaluates only the final predicted class, it ignores the predicted probabilities, which contain richer information.

To evaluate probability quality, define the indicator variable:

$\delta_{ij} =
\begin{cases}
1 & \text{if } y_i = c_j \
0 & \text{otherwise}
\end{cases}$

This indicator encodes the true class in a one-hot representation.

The classification error for class $j$ at observation $i$ is:

$e_{ij} = \delta_{ij} – \hat p_{ij}$

This error measures how far the predicted probability is from the true class indicator.

---

Root Average Squared Error (RASE) for Nominal Classification

To summarize probability errors across all observations and all classes, define the Root Average Squared Error (RASE) as:

$\text{RASE} = \sqrt{
\frac{1}{nk}
\sum_{i=1}^n
\sum_{j=1}^k
(\delta_{ij} – \hat p_{ij})^2
}$

Interpretation:

• RASE measures how well the predicted probability vectors match the true class indicators.
• It penalizes confident but incorrect predictions more heavily than uncertain ones.
• It incorporates all classes, not only the predicted class.

Lower RASE values indicate better probabilistic classification performance.

---

RASE in the Ideal Case

In the ideal classification scenario, the model assigns:

• Probability 1 to the true class $c_s$,
• Probability 0 to all other classes.

Formally:

• $\hat p_{is} = 1$ when $\delta_{is} = 1$,
• $\hat p_{ij} = 0$ for all $j \ne s$.

Thus:

$\delta_{ij} = \hat p_{ij} \quad \text{for all } i, j$

All squared differences are zero, and therefore:

$\text{RASE} = 0$

This is the minimum possible value, corresponding to perfect classification with complete certainty.

---

RASE in the Worst Case

In the worst-case scenario, the model assigns probability 1 to the wrong class $c_t$ where $t \ne s$.

For a given observation:

• $\delta_{is} = 1$ but $\hat p_{is} = 0$,
• $\delta_{it} = 0$ but $\hat p_{it} = 1$,
• all other terms are zero.

The sum of squared errors for that observation becomes:

$\sum_{j=1}^k (\delta_{ij} – \hat p_{ij})^2 = 2$

Averaging across all classes yields the maximum RASE:

$\text{RASE}_{\max} = \sqrt{\frac{2}{k}}$

This corresponds to confident but completely incorrect predictions.

---

RASE in the Inclusive (Uniform) Case

In the inclusive case, the model assigns equal probability to all classes:

$\hat p_{ij} = \frac{1}{k} \quad \text{for all } j$

There is no dominant predicted class.

For the true class $c_s$:

$\delta_{is} – \hat p_{is} = 1 – \frac{1}{k}$

For all other classes $j \ne s$:

$\delta_{ij} – \hat p_{ij} = -\frac{1}{k}$

Summing squared errors across all classes gives:

$\sum_{j=1}^k (\delta_{ij} – \hat p_{ij})^2 = \frac{k-1}{k}$

Thus, the RASE in this case is:

$\text{RASE} = \sqrt{\frac{k – 1}{k}}$

This value corresponds to a completely non-informative classifier.

---

Interpretation and Preferred Range of RASE

From the previous cases, the RASE satisfies:

$0 \le \text{RASE} \le \sqrt{\frac{2}{k}}$

Important benchmark:

• $\sqrt{\frac{k – 1}{k}}$ corresponds to uniform guessing.
• A useful classification model should achieve:

$\text{RASE} < \sqrt{\frac{k – 1}{k}}$

This indicates that the model provides more information than random or uniform assignment.

---

Key Takeaways

• Nominal classification models produce both predicted classes and predicted probabilities.
• Misclassification rate and accuracy evaluate hard decisions.
• RASE evaluates probabilistic quality across all classes.
• RASE = 0 represents perfect classification.
• RASE = $\sqrt{(k-1)/k}$ represents no information.
• RASE = $\sqrt{2/k}$ represents confidently wrong predictions.
• A good model must beat the uniform probability baseline.

Together, these metrics provide a complete picture of nominal classification performance, capturing both decision correctness and probability calibration.