Binary Cross-Entropy (BCE)

Binary cross-entropy — log loss — scores predicted probabilities against binary labels. Over \(n\) rows:

\[ L = -\frac{1}{n} \sum_{i=1}^{n} \Big[ y_i \log(\hat{p}_i) + (1 - y_i)\log(1 - \hat{p}_i) \Big] \]

Only one term survives per row. Where the label is 1 the loss is \(-\log(\hat{p})\); where it is 0 it is \(-\log(1-\hat{p})\). Either way, being closer to the truth costs less.

Two properties, one of them the reason it exists

import numpy as np

print("  the loss on a single row, when the true label is 1")
for p in (0.99, 0.90, 0.50, 0.10, 0.01, 0.001):
    print(f"    predicted {p:>6.3f}   loss {-np.log(p):>8.4f}")

rng = np.random.default_rng(0)
truth = 0.30
y = (rng.random(200_000) < truth).astype(int)
"""The events happen 30% of the time. What constant probability should a forecaster
   report to minimize the average loss? Reporting anything but the truth costs."""
print(f"\n  average loss over {len(y):,} rows that occur {y.mean():.4f} of the time")
for p in (0.10, 0.20, 0.30, 0.40, 0.60):
    loss = -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))
    print(f"    reporting {p:.2f} every time   {loss:.6f}")

print("\n  one confidently wrong row among a thousand good ones")
good = np.full(1000, 0.99)
for wrong in (0.5, 0.1, 0.01, 0.0001):
    batch = np.append(good, wrong)
    labels = np.ones(len(batch))
    print(f"    the bad row predicted {wrong:<7} average loss {-np.mean(np.log(batch)):.6f}")
#   the loss on a single row, when the true label is 1
#     predicted  0.990   loss   0.0101
#     predicted  0.900   loss   0.1054
#     predicted  0.500   loss   0.6931
#     predicted  0.100   loss   2.3026
#     predicted  0.010   loss   4.6052
#     predicted  0.001   loss   6.9078
#
#   average loss over 200,000 rows that occur 0.2998 of the time
#     reporting 0.10 every time   0.764055
#     reporting 0.20 every time   0.638734
#     reporting 0.30 every time   0.610682
#     reporting 0.40 every time   0.632378
#     reporting 0.60 every time   0.794738
#
#   one confidently wrong row among a thousand good ones
#     the bad row predicted 0.5     average loss 0.010733
#     the bad row predicted 0.1     average loss 0.012341
#     the bad row predicted 0.01    average loss 0.014641
#     the bad row predicted 0.0001  average loss 0.019241

The penalty has no ceiling. Predicting 0.99 when the label is 1 costs 0.0101; predicting 0.001 costs 6.9078, and as the prediction approaches zero the loss approaches infinity. The third block shows what that means in aggregate — a single row out of a thousand, predicted at 0.0001, takes the average loss from 0.010733 to 0.019241. One row nearly doubles the score.

That is a design choice with consequences. It makes the loss responsive to exactly the errors that matter most in a risk model, and it makes the loss fragile when labels are wrong — a mislabeled row that the model confidently and correctly disagrees with is punished without limit, which is why label noise work reaches for bounded losses instead.

It is minimized by telling the truth. The middle block is the property that makes cross-entropy more than a convenient differentiable function. On events that occur 30% of the time, reporting 0.30 gives the lowest average loss — 0.610682, against 0.638734 for 0.20 and 0.632378 for 0.40. Overstating and understating both cost, and the minimum sits exactly at the true rate.

This population-optimal property makes cross-entropy a proper scoring rule. It does not guarantee that a fitted model is calibrated: its function class, regularization, finite sample and optimization can prevent it from attaining the true conditional probabilities. Logistic regression often provides a useful probability baseline, but it still needs representative calibration checks.

The Brier score is the other common proper scoring rule, with a bounded quadratic penalty rather than an unbounded logarithmic one. Which to prefer depends on whether confident errors should dominate the score.

Practical notes

• Handle probability endpoints explicitly. A wrong certain prediction has infinite ideal log loss; a correct endpoint prediction has loss zero by continuity. Implementations may clip probabilities to avoid undefined numerical expressions such as log(0), so record the convention when comparing extreme-probability results. A fused logit-based loss avoids forming those probabilities during training.
• The value is not interpretable alone. A log loss of 0.61 is excellent on a balanced problem and terrible on one with a 1% base rate — compare it against the loss of predicting the base rate for every row.
• Sigmoid plus cross-entropy is one operation. Frameworks provide a fused version because computing them separately overflows; the separate form is a common source of instability in hand-rolled training loops.
• Class weights change what is optimal. Weighting the loss makes reporting the true probability no longer the minimum — which is precisely the trade class weighting makes, and why it costs calibration.

Minimizing cross-entropy is equivalent to minimizing the Kullback–Leibler divergence between the true label distribution and the predicted one — which is the information-theoretic statement of the same thing the middle block measures.

References

• scikit-learn API reference, log_loss
• scikit-learn User Guide, Probability calibration