1. Definition

  • In classification, instead of just predicting a class label, a model can output probabilities for each possible class.
  • A classification probability is the model’s estimate of how likely each class is given the input features.

Formally:

$P(Y = k \mid X = x), \quad k = 1,2,\dots,K$

where $K$ = number of classes.

2. Example

Binary Classification (Spam Detection)

  • Email → classify as Spam or Not Spam.
  • Model outputs:
    • $P(\text{Spam} \mid x) = 0.85$
    • $P(\text{Not Spam} \mid x) = 0.15$
  • Predicted label = Spam (since 0.85 > 0.5).

Multiclass Classification (Digit Recognition, 0–9)

  • Image of a digit → model outputs:
    • {0: 0.01, 1: 0.02, 2: 0.92, …, 9: 0.00}
  • Predicted label = 2 (highest probability).

3. How Models Produce Probabilities

  • Logistic Regression: outputs probability via sigmoid (binary).
  • Softmax Regression / Neural Networks: outputs probability distribution across classes.
    • $P(y=k|x) = \frac{e^{z_k}}{\sum_j e^{z_j}}$
  • Naïve Bayes: applies Bayes’ rule to compute class probabilities.
  • Tree Ensembles (Random Forest, Gradient Boosting): probability = fraction of trees voting for a class (or calibrated via Platt scaling / isotonic regression).

4. Why Probabilities Are Useful

  • More informative than labels:
    • Label: “Spam.”
    • Probability: “85% Spam, 15% Not Spam.”
  • Support decision-making with thresholds (e.g., block only if probability > 0.9).
  • Allow ranking (e.g., fraud detection: review top 5% highest-risk transactions).
  • Needed for probabilistic evaluation metrics (Log Loss, Brier Score).

5. Evaluation of Classification Probabilities

  • Accuracy only uses predicted labels (hard decisions).
  • But probabilities require proper scoring rules:
    • Log Loss (Cross-Entropy): penalizes confident wrong predictions.
    • Brier Score: squared error between predicted probability and outcome.
    • Calibration curves: check if predicted probabilities match observed frequencies (e.g., “70% rain” should rain ~70% of the time).

6. Example: Weather Forecasting

  • Deterministic forecast: “It will rain.”
  • Classification probabilities:
    • $P(\text{Rain}) = 0.7$
    • $P(\text{No Rain}) = 0.3$
  • Allows decisions:
    • Carry an umbrella if threshold > 0.6.

Summary:
Classification probabilities are the predicted likelihoods assigned to each class by a classification model. They are produced by models like logistic regression, softmax neural nets, Naïve Bayes, and ensemble trees, and are evaluated using metrics like log loss, Brier score, and calibration plots. They provide richer information than just class labels, making them essential for risk-sensitive decision-making.