Log Loss

Log loss scores probability forecasts by the logarithm of the probability assigned to what actually happened. For binary outcomes,

\[ \text{log loss} = -\frac{1}{N}\sum_{i=1}^{N}\Big[ y_i \log \hat{p}_i + (1 - y_i)\log(1 - \hat{p}_i) \Big] \]

Lower is better, and zero is only reached by predicting 1 for every event that happens and 0 for every one that does not. It is the same quantity as cross-entropy and as the negative log-likelihood, which is why so many models minimize it during training: minimizing log loss is maximum likelihood estimation, not an approximation to it.

Why it grows without bound

The defining behavior is what happens as a prediction becomes confidently wrong. The Brier score charges a squared error, which cannot exceed 1 no matter how wrong the forecast is. Log loss has no such ceiling.

import numpy as np

def log_loss(y, p):
    return float(-np.mean(y * np.log(p) + (1 - y) * np.log(1 - p)))

def brier(y, p):
    return float(np.mean((p - y) ** 2))

y = np.array([1, 0, 1])
p = np.array([0.9, 0.2, 0.1])
print(f"{log_loss(y, p):.4f} {brier(y, p):.4f}")
# 0.8770 0.2867

confidences = (0.5, 0.1, 0.01, 0.001, 1e-6)
# The penalty each rule charges for one prediction that is wrong, as it grows more confident.
for predicted in confidences:
    single = np.array([1])
    print(f"{predicted:<8} {log_loss(single, np.array([predicted])):>9.4f} "
          f"{brier(single, np.array([predicted])):.6f}")
# 0.5 0.6931 0.250000
# 0.1 2.3026 0.810000
# 0.01 4.6052 0.980100
# 0.001 6.9078 0.998001
# 1e-06 13.8155 0.999998

Brier saturates: past about 0.01 there is almost nothing left to charge, so a forecast that is certain and wrong costs barely more than one that was merely very wrong. Log loss keeps going, roughly doubling each time the probability drops by a factor of a hundred. That is the property to want when confident mistakes are genuinely worse than uncertain ones — a medical or credit decision — and the property to watch when they are not.

The limit case is not an edge case. Predicting exactly 0 for an event that occurs gives an infinite loss, which would make any average infinite, so implementations clip the probabilities first. scikit-learn’s log_loss returns 36.0437 for a single such prediction — and that number is \(-\log(\varepsilon)\) for the floating-point epsilon, not a property of the metric. A reported log loss dominated by a handful of clipped predictions is reporting the library’s constant. Clipping probabilities into a sane range yourself, and saying where, is better than discovering it later.

A worked example

The average is 0.877, and one prediction supplies 88 percent of it. That concentration is normal and is the main thing to know when reading a log loss: the number is often a statement about a few rows rather than about the model as a whole. Checking the distribution of per-row contributions, not just their mean, is what separates a genuinely poor model from one with a handful of mislabeled records.

What it shares with Brier, and what it does not

Both are strictly proper scoring rules: the expected loss is minimized uniquely by reporting your true probability, so neither can be improved by shading the output. That is the reason to use either as a model-selection metric rather than accuracy — see strictly proper scoring rules, where the property is demonstrated by calculation.

Neither travels between datasets. Part of the score is the irreducible uncertainty of the outcome, which depends on the base rate, so a log loss of 0.05 on a rare-event problem and 0.5 on a balanced one say nothing about which model is better. The comparison that means something is against a constant forecast of the base rate on the same data.

Beyond two classes

For \(K\) classes the form generalizes to \(-\frac{1}{N}\sum_i \log \hat{p}_{i, y_i}\): the log of the probability the model gave to the class that actually occurred, averaged. The binary formula is the same thing written out for two classes. Only the probability assigned to the true class enters, which means a model can redistribute probability freely among the wrong classes without changing its score — a property worth knowing when the wrong classes are not equally wrong.

Because log loss rewards calibration as well as ranking, a model can lower it without changing a single decision, by fixing the probability scale alone — see Platt scaling and model calibration.

References

• scikit-learn documentation, sklearn.metrics.log_loss
• Tilmann Gneiting and Adrian E. Raftery, “Strictly Proper Scoring Rules, Prediction, and Estimation”, Journal of the American Statistical Association 102(477), 359–378, 2007