Probability and Information Theory for Deep Learning

The probability vocabulary is developed in Probability for Machine Learning. Here we recall the parts needed to connect a network’s predictions to likelihood, cross-entropy, and KL divergence. If those basics are familiar, start at maximum likelihood.

A binary classifier might assign probability 0.9 to the label that occurs, while another assigns 0.1. Negative log-likelihood gives those predictions losses of about 0.105 and 2.303. This article explains that calculation and connects it to entropy and KL divergence. Networks can also output real-valued predictions or representations, and losses can be chosen directly for the errors they penalize; a probabilistic interpretation is useful when it matches the task.

Random variables and distributions

A random variable assigns a numerical value to an uncertain outcome. A discrete variable takes countably many possible values, with probabilities \(P(X=x)\) that sum to one. For a continuous variable with density \(p\), interval probabilities are areas: \(P(a\le X\le b)=\int_a^b p(x)\,dx\). A single point has probability zero. The density itself can exceed 1: a uniform density of 2 on \([0,0.5]\) has total area \(2\times0.5=1\).

Three distributions recur in the loss functions and latent-variable models discussed here.

Bernoulli describes one binary outcome, encoded as 0 or 1, with probability \(p\) of 1. A sigmoid maps a real network output to a number between 0 and 1; interpreted probabilistically, that number is the model’s Bernoulli parameter. How well that parameter estimates the true probability depends on the data and fitted model.

\[P(X=1)=p,\qquad P(X=0)=1-p\]

Gaussian (normal) has mean \(\mu\) and variance \(\sigma^2>0\). It is one common model for continuous noise and one option for weight initialization. A standard Gaussian is also a common prior (a distribution specified before observing an example) for the latent vector in a variational autoencoder. That vector represents the example through unobserved numerical features. For a scalar Gaussian, the density is:

\[p(x)=\frac{1}{\sqrt{2\pi\sigma^{2}}}\exp\left(-\frac{(x-\mu)^{2}}{2\sigma^{2}}\right)\]

The categorical distribution describes one of \(C\) mutually exclusive classes, with probabilities \(p_1,\ldots,p_C\ge0\) summing to one. A softmax converts a vector of real scores into such a probability vector. For three classes, \((0.7,0.2,0.1)\) assigns probability 0.2 to the second class. Separate sigmoid outputs instead allow several labels to be present at once.

Choosing an objective for a problem is the subject of Loss Function Design: Choosing an Objective That Matches the Problem.

Expectation and variance

For a variable with finite expectation, \(\mathbb{E}[X]\) is its probability-weighted average. If its second moment \(\mathbb{E}[X^2]\) is finite, variance \(\operatorname{Var}(X)=\mathbb{E}[(X-\mathbb{E}[X])^2]\) measures spread; its square root is the standard deviation.

\[\mathbb{E}[X]=\sum_{x}xP(x)\qquad\text{or}\qquad\int x\,p(x)\,dx\]

The notation \(\mathbb{E}_{x\sim p}[f(x)]\) means averaging \(f(x)\) under distribution \(p\). For example, a Bernoulli variable with \(p=0.8\) has expectation \(0\times0.2+1\times0.8=0.8\), even though each outcome is 0 or 1. A sample average estimates an expectation; it is not the population quantity itself. In training, the averaging may be over data, augmentation, dropout masks, or latent samples. Read the subscript to identify which randomness is being averaged.

import numpy as np

rng = np.random.default_rng(0)
x = rng.normal(loc=2.0, scale=3.0, size=100_000)

print(round(x.mean(), 3), round(x.var(), 3), round(x.std(), 3))
# 1.997 9.002 3.0
print(round((3 * x + 5).mean(), 3), round(3 * x.mean() + 5, 3))
# 10.992 10.992
print(round((3 * x).var(), 3), round(9 * x.var(), 3))
# 81.021 81.021

The first output estimates the population mean 2, variance 9, and standard deviation 3 from one sample. The second line checks the sample counterpart of linearity, \(\mathbb{E}[3X+5]=3\mathbb{E}[X]+5\). Multiplying the observations by 3 multiplies their variance by 9, as the last line checks: \(\operatorname{Var}(3X)=9\operatorname{Var}(X)\). Xavier and He initialization use variance scaling together with assumptions about sums of weighted inputs; converting a chosen variance into a standard deviation introduces the square root.

Conditional probability and independence

A probabilistic classifier estimates the distribution of labels given an input: \(P_\theta(Y=y\mid X=x)\). Here \(\theta\) denotes its learned weights and biases. The conditional bar means that the input is held fixed while the possible labels vary. Some classifiers produce only scores or decisions; other neural networks model a joint distribution or a continuous target instead.

Events \(A\) and \(B\) are independent when \(P(A\cap B)=P(A)P(B)\). For the supervised likelihood below, assume labels from different examples are independent conditional on their inputs and the parameters, and use the same conditional model for each example. I.i.d. input-label pairs are a common setup that gives this factorization. It does not mean that a label is independent of its own input. Dependent data need a joint model or an appropriate conditional factorization.

Maximum likelihood and prediction losses

For a dataset of \(m\) input-label pairs \((x^{(i)},y^{(i)})\), maximum likelihood chooses parameters that maximize the product of the probabilities assigned to the observed labels. The superscript \((i)\) indexes examples. The notation \(\arg\max\) asks for the parameters attaining the largest value, when such a maximizer exists:

\[\theta^{*}=\arg\max_{\theta}\prod_{i=1}^{m}P_\theta(y^{(i)}\mid x^{(i)})\]

Multiplying many small probabilities can underflow to zero in floating-point arithmetic. Compute their log-probabilities and add them instead of taking a log after forming the product. The logarithm is strictly increasing, so it preserves the maximizers of a positive likelihood. Negating the sum gives the negative log-likelihood (NLL), which is minimized. All logarithms here are natural logs:

\[\theta^{*}=\arg\min_{\theta}\;-\sum_{i=1}^{m}\log P_\theta(y^{(i)}\mid x^{(i)})\]

Dividing the NLL by the fixed number of examples gives a mean loss with the same minimizers. For continuous targets, likelihood uses the conditional density \(p_\theta(y\mid x)\), not the probability of the exact observed value. Its log-density can be positive, so a continuous NLL can be negative.

For a binary label \(y\in\{0,1\}\), the probability of the observed outcome is \(p^y(1-p)^{1-y}\). For \(0

For a Gaussian target with predicted mean \(\mu_\theta(x)\) and one fixed variance \(\sigma^2\) shared across examples, the per-example NLL is \((y-\mu_\theta(x))^2/(2\sigma^2)+\log\sqrt{2\pi\sigma^2}\). Thus minimizing the mean NLL over the mean parameters is equivalent to minimizing MSE. If the model also learns a variance, that scale and the log-variance term must remain in the objective. Exercise 1 works through the fixed-variance case.

def bernoulli_nll(p, y):
    """Binary NLL evaluated after clipping the predicted probability."""
    p = np.clip(p, 1e-12, 1 - 1e-12)
    return -(y * np.log(p) + (1 - y) * np.log(1 - p))

print(round(bernoulli_nll(0.9, 1), 4))
# 0.1054
print(round(bernoulli_nll(0.5, 1), 4))
# 0.6931
print(round(bernoulli_nll(0.1, 1), 4))
# 2.3026

For the observed label 1, assigning probability 0.9 gives loss 0.1054, while assigning 0.1 gives 2.3026. The model prediction \(p=0.5\) gives \(\log2\approx0.6931\); whether that prediction is useful depends on the base rate and other available predictions. The theoretical loss grows without bound as the assigned probability of the observed outcome approaches zero. This demonstration clips probabilities to avoid evaluating \(\log0\), which caps that loss. Training implementations commonly compute binary cross-entropy directly from logits (the real scores before sigmoid) for numerical stability.

Information theory in three quantities

For the finite discrete distributions in this section, the information content of an outcome with positive probability is \(-\log P(x)\). An outcome with probability \(1/2\) has information \(\log2\); one with probability \(1/8\) has \(3\log2\). Entropy averages this quantity over outcomes. Natural logarithms measure it in nats; base-2 logarithms would give bits:

\[H(P)=-\sum_{x}P(x)\log P(x)\]

Cross-entropy is the expected negative log-probability assigned by \(Q\) when outcomes follow \(P\). In an idealized coding interpretation, it is the average description length using probabilities from \(Q\), with the logarithm base determining the unit:

\[H(P,Q)=-\sum_{x}P(x)\log Q(x)\]

KL divergence is the excess expected log loss from using \(Q\) in place of \(P\). Here both are distributions over the same categories; \(P\) may be a population distribution or a specified reference:

\[D_{KL}(P\Vert Q)=H(P,Q)-H(P)=\sum_{x}P(x)\log\frac{P(x)}{Q(x)}\]

We use \(0\log0=0\). If \(P(x)>0\) but \(Q(x)=0\), cross-entropy and \(D_{KL}(P\Vert Q)\) are infinite. The code uses SciPy’s rel_entr and xlogy to handle these boundary values without adding an epsilon that changes the distributions. Inputs here are nonnegative probability vectors that sum to one.

from scipy.special import xlogy, rel_entr

def entropy(p):
    return -np.sum(xlogy(p, p))

def cross_entropy(p, q):
    return -np.sum(xlogy(p, q))

def kl(p, q):
    return np.sum(rel_entr(p, q))

P = np.array([0.7, 0.2, 0.1])
Q = np.array([0.5, 0.3, 0.2])
print(round(entropy(P), 4), round(cross_entropy(P, Q), 4), round(kl(P, Q), 4))
# 0.8018 0.8869 0.0851 # cross-entropy = entropy + KL
print(round(entropy(np.array([1/3, 1/3, 1/3])), 4))
# 1.0986
print(round(kl(P, P), 4))
# 0.0
print(round(kl(P, Q), 4), round(kl(Q, P), 4))
# 0.0851 0.092
print(kl(np.array([0.9, 0.1]), np.array([1.0, 0.0])))
# inf

The output separates the reference entropy, 0.8018 nats, from the expected log loss under \(Q\), 0.8869 nats. Their difference, 0.0851, is \(D_{KL}(P\Vert Q)\). With \(P\) fixed, minimizing cross-entropy over \(Q\) also minimizes this KL divergence. KL is nonnegative and is zero exactly when the distributions match, but exchanging its arguments can change its value.

For a labeled example of class \(c\), take the target distribution to be one-hot: it assigns 1 to \(c\) and 0 to the other classes. Then \(H(P,Q)=-\log Q(c)\), which connects the information-theory formula to the categorical NLL above. Averaging over training examples gives the empirical classification loss; averaging over the actual input-label distribution would give its population counterpart.

For \(C\) categories, entropy is largest at the uniform distribution and equals \(\log C\). A classifier predicting exactly \(1/C\) for every class also has cross-entropy \(\log C\) for any observed label. This is a baseline for nearly uniform outputs, not a requirement on every untrained network: random logits need not be nearly equal. For example, a two-class model that always predicts \((0.9,0.1)\) has average loss \(-\tfrac12(\log0.9+\log0.1)\approx1.204\) on balanced labels, above \(\log2\).

Sampling, and why seeds matter

Drawing a value from a distribution is sampling, written \(x\sim p\). Initialization, dropout, augmentation, and generation use random draws; shuffling randomly permutes the training examples. Reusing a seed reproduces a generator’s sequence when the generator, environment, and sequence of calls are the same. A seed alone does not guarantee identical training results across devices, library versions, or nondeterministic operations.

For \(n\) independent, identically distributed scalar draws with finite variance \(\sigma^2\), the sample mean has variance \(\sigma^2/n\) and standard deviation \(\sigma/\sqrt n\). For gradients evaluated at fixed parameters, the same rule applies to each component when examples are sampled independently with replacement from a fixed dataset and the loss is an average of per-example losses. The average gradient then estimates the full training-set gradient. Without-replacement sampling introduces a finite-population correction (the variance reaches zero when a batch contains the whole dataset); dependence or operations that couple examples, such as training-mode batch normalization, change this simple model.

true_mean = 2.0
for n in [10, 100, 1000, 10_000]:
    means = [rng.normal(true_mean, 3.0, n).mean() for _ in range(400)]
    print(n, round(float(np.std(means)), 4), round(3.0 / np.sqrt(n), 4))
# 10 0.963 0.9487
# 100 0.2958 0.3
# 1000 0.0984 0.0949
# 10000 0.0322 0.03

Each row above summarizes 400 independently drawn batch means: the second number is their measured standard deviation, and the third is \(3/\sqrt n\). The differences reflect the finite number of repetitions. This experiment illustrates the sampling rule for Gaussian values; it does not measure neural-network gradients or training speed.

The multiclass output layer is covered in Softmax and Multiclass Classification Explained.

Using the probabilistic interpretation

These calculations require probability distributions, expectations, and logarithms. Statistical inference becomes relevant when evaluating uncertainty in measured model performance, while latent-variable models require further conditional-probability reasoning. Those are extensions to this setup, not prerequisites for the loss calculations above.

A likelihood interpretation connects an objective to a model of the targets: Bernoulli and categorical outputs give classification log losses, while a fixed-variance Gaussian gives an objective equivalent to MSE. Choosing MSE does not by itself assert Gaussian data; squared error can also be selected because large errors deserve larger penalties, and its population minimizer is the conditional mean when the relevant second moments are finite. Loss Function Design considers these choices in context.

The latent-variable formulation is developed in Variational Autoencoders: ELBO, Reparameterization, and Latent Space.

Exercises

1. Derive a loss. Assume the model outputs \(\mu\) and the data is Gaussian with fixed variance \(\sigma^2\). Write the negative log-likelihood of one observation and simplify. Which familiar loss appears, and what happened to \(\sigma\)?

You should get: a squared term plus additive constants. The constants do not affect the argmin.

2. KL is not symmetric. Take \(P=[0.9,0.1]\) and \(Q=[0.5,0.5]\). Compute \(D_{KL}(P\Vert Q)\) and \(D_{KL}(Q\Vert P)\). Then set \(Q=[0.99,0.01]\) and repeat. Then consider \(Q=[1,0]\): which direction becomes infinite, and why?

You should get: two different numbers in each case, and the two finite cases order the directions oppositely; the boundary case tests which distribution weights the sum.

import numpy as np
from scipy.special import rel_entr
kl = lambda p, q: np.sum(rel_entr(p, q))
P=np.array([.9,.1])
for Q in (np.array([.5,.5]), np.array([.99,.01])):
    print(np.round([kl(P,Q), kl(Q,P)],4))
# [0.3681 0.5108]
# [0.1445 0.0713]

Q = np.array([1.0, 0.0])
print(kl(P, Q), round(kl(Q, P), 4))
# inf 0.1054

3. Batch size and noise. Empirically confirm that the standard deviation of a batch mean falls as \(\sigma/\sqrt{n}\). Then answer: under the independent-sampling assumptions above, by what factor must you increase batch size to halve the standard deviation of a gradient component? Distinguish sample-processing work per update from wall-clock time and total training cost.

You should get: a factor of four; the sampling formula alone does not determine the fastest training configuration.