Loss Functions

A loss function turns a prediction and a target into a number that training seeks to reduce. The optimization method must be able to work with its structure: smooth losses provide gradients, while losses such as absolute error and hinge loss can use subgradients at their nondifferentiable points. Match the loss to the quantity the task requires.

Three terms get used interchangeably and are not the same. The loss is the error on one sample, the cost is the average over the batch or dataset, and the objective is what the optimizer actually minimizes — usually the cost plus a regularization term.

The usual descriptions of the regression losses — squared error punishes large errors, absolute error is robust to outliers — are true and undersell what is happening.

The loss decides what gets estimated

import numpy as np
from scipy.optimize import minimize_scalar

rng = np.random.default_rng(0)
clean = rng.normal(100, 10, size=500)
"""What single number best summarizes these values? The answer depends entirely
   on which loss you are minimizing, not on the data."""

def fit(values, loss):
    return minimize_scalar(lambda c: loss(values - c), bounds=(0, 500), method="bounded").x

losses = {
    "squared error": lambda r: np.mean(r ** 2),
    "absolute error": lambda r: np.mean(np.abs(r)),
    "Huber (delta=5)": lambda r: np.mean(np.where(np.abs(r) <= 5, 0.5 * r ** 2, 5 * (np.abs(r) - 2.5))),
}
print(f"{'contaminating values at 1000':>29}  " + "  ".join(f"{k:>16}" for k in losses))
for k in (0, 1, 5, 25):
    data = np.concatenate([clean, np.full(k, 1000.0)])
    print(f"{k:>29}  " + "  ".join(f"{fit(data, f):>16.4f}" for f in losses.values()))
print(f"\n  the clean data's mean is {clean.mean():.4f} and its median is {np.median(clean):.4f}")
print("  squared error chases the mean, absolute error stays at the median,")
print("  Huber is quadratic for small residuals and limits the influence of large ones")
#  contaminating values at 1000     squared error    absolute error   Huber (delta=5)
#                             0           99.7311           99.4987           99.5437
#                             1          101.5280           99.5020           99.5698
#                             5          108.6447           99.6117           99.6754
#                            25          142.6010          100.0221          100.2136
#
#   the clean data's mean is 99.7311 and its median is 99.4980
#   squared error chases the mean, absolute error stays at the median,
#   Huber is quadratic for small residuals and limits the influence of large ones

On clean data the three losses nearly agree: 99.73, 99.50, 99.54, which are the mean, the median and something between them.

Add twenty-five contaminating values and the squared-error fit is dragged to 142.60 while absolute error sits at 100.02 and Huber at 100.21. The data underneath has not changed — five hundred observations still cluster around 100 — but one of these three summaries has left the neighborhood.

That is because minimizing squared error estimates the mean and minimizing absolute error estimates the median, and the mean is not robust. The choice of loss is therefore a choice about which number you want, not only about how errors are weighted — the principle is set out with its source in regression models, and the table above is what it costs.

Huber interpolates deliberately: quadratic while the residual is within \(\delta\), linear beyond it. The threshold \(\delta\) controls where the slope stops growing with residual size. Large residuals still contribute, but their influence on the fitted location is bounded; the threshold does not determine whether an observation is genuine signal or an outlier.

The practical consequence is that a model trained on squared error aims at the mean, while absolute error is minimized by the median, so judging it on absolute error measures it against a quantity it was not aiming at. That is not always a mistake: the training loss is often a convenient surrogate, and the reported metric should be the one that matches the decision. Compare models on that decision metric, and keep in mind which quantity each loss targets when reading the comparison.

Matching the loss to the output layer

• Binary: a single logit into a binary cross-entropy that applies the sigmoid itself. The cross-entropy article covers why it is minimized by reporting the true probability.
• Multi-class: \(K\) logits into a categorical cross-entropy that applies the softmax itself. Applying softmax first and then the loss is a common bug — PyTorch’s CrossEntropyLoss expects raw logits.
• Multi-label: one logit per label into independent binary cross-entropies, because the labels are not competing for one unit of probability.
• Margin-based: hinge loss, \(\max(0, 1 - y\hat{y})\), which is what an SVM minimizes and which cares only that the correct class clears a margin, not about probabilities at all.

For ranking and representation tasks, contrastive and triplet losses replace “match the label” with “keep similar things closer than dissimilar ones,” which is a different kind of objective again. Instead of an absolute target for each individual prediction, they learn relations between examples, such as positive and negative pairs or anchor–positive–negative triplets.

Reading a loss value

A loss is a training signal, not a report. Its units depend on the target’s units for regression and on the base rate for classification, so a cross-entropy of 0.61 improves on the constant base-rate baseline of about 0.693 for balanced binary labels but is worse than the corresponding baseline of about 0.056 for a 1% positive rate, and a mean squared error of 400 means nothing without knowing the scale of the outcome.

Two comparisons make it interpretable: against a trivial baseline — the loss of always predicting the mean, or the base rate — and against the same loss on held-out data computed under the same evaluation conditions, where a large gap is a clue to overfitting. Check first that the two numbers are comparable: a different data distribution, or a training-only penalty or augmentation in one of them, can open a gap without overfitting. Reporting the training loss alone tells the reader neither.

And the loss is rarely the thing anyone cares about. Cross-entropy is minimized by honest probabilities; the business cares about decisions at a threshold. Keeping both in the report — the loss the model optimized and the metric the decision uses — avoids the situation where training improves and nothing else does.

References

• PyTorch documentation, CrossEntropyLoss
• PyTorch documentation, HuberLoss
• scikit-learn API reference, HuberRegressor