Bayesian Neural Networks (BNNs)

A Bayesian neural network treats its weights as random variables rather than fixed numbers. Instead of one trained network you get a posterior distribution over networks, \(p(W \mid D) \propto p(D \mid W)\,p(W)\), and predictions come from averaging over it: \(p(y \mid x, D) = \int p(y \mid x, W)\, p(W \mid D)\, dW\).

A predictor that returns only a point estimate does not report how uncertain that estimate is. One way to add model uncertainty is to place a distribution over weights. Fixed-weight systems can also return probabilistic outputs or other uncertainty scores; the question is what uncertainty each construction represents and how it is validated.

From a prior to a prediction

W denotes the network’s parameters and D the observed training data. The prior p(W) assigns plausibility before observing D. The likelihood p(D | W) says how probable the observations are under a particular network and observation model. Their product, normalized to integrate to one, defines the posterior p(W | D). A Gaussian regression likelihood and a categorical classification likelihood encode different assumptions about the outputs; choosing a distribution over weights does not specify the likelihood for you.

Posterior prediction averages the output distribution from each plausible parameter setting, weighted by its posterior probability. In practice, one can sample several parameter settings from a posterior approximation, run the input through each network, and average the resulting predictive distributions. This is not generally the same as inserting the mean weights into one network, because its output is nonlinear in those weights.

Separating uncertain parameters from noisy outcomes

Suppose a regression network with parameters W predicts mean fW(x) and observation variance σ²W(x). The posterior predictive variance decomposes into the posterior mean of σ²W(x) plus the posterior variance of fW(x). The first describes outcome variation within a parameter setting; the second describes disagreement over the predicted mean. They are often called aleatoric and epistemic contributions, but their interpretation is conditional on the model and inference assumptions.

For a small hypothetical posterior approximation with two equally weighted predictions, let the means be 0 and 2 and the observation variances both 1. The predictive mean is 1. Average observation variance is 1 and variance of the two means around 1 is 1, giving total predictive variance 2. Reporting only the spread of the means would omit observation noise. Reporting only one network’s variance would omit disagreement about the mean.

A Bayesian final layer offers a tractable special case. Fix a feature vector h(x) produced by earlier layers, put a Gaussian prior on output weights, and use Gaussian observation noise in a linear output layer. The posterior output weights are Gaussian with mean m and covariance S. The predictive mean is h(x)ᵀm and its variance is h(x)ᵀSh(x) + σ². The first term projects weight uncertainty onto this input’s feature direction; the second adds observation noise. This is a Bayesian output layer, not posterior inference over the fixed feature extractor.

Why the prior and representation matter

Data constrains combinations of weights that affect observed predictions. Directions poorly constrained by the observations remain influenced by the prior. A tight prior can report little uncertainty even in regions without data, while a wider prior may permit more variation. Distance in raw input space alone does not determine uncertainty: the feature map and prior determine which function behaviors are considered related.

Under the Gaussian linear-output assumptions above, the posterior mean coincides with a ridge-regression estimate for a matching penalty. That shows why a more stable point prediction need not be a benefit unique to Bayesian averaging; regularization can produce the same mean. The covariance adds a different object, a distribution expressing uncertainty under the assumed model. Neither object guarantees reliable behavior under misspecification.

Check predictive calibration on held-out data, including interval coverage and width for regression, or probability scores and reliability for classification. Consider relevant distribution changes separately. A nominal 95% model-based predictive interval is not an unconditional guarantee that 95% of outcomes in a new population will fall within it. Prior sensitivity and comparison with simpler baselines help reveal which conclusions depend on modeling choices.

Why the integral is the problem

The final-layer closed form works because its likelihood and prior form a Gaussian linear model. Making hidden weights random generally makes exact normalization and posterior prediction computationally difficult, even though the posterior can still be specified as likelihood times prior. Practical methods approximate the required calculations:

• Variational inference fits a tractable distribution q(W), often by balancing expected log likelihood against a KL penalty measuring departure from the prior. This is the evidence lower bound objective under the usual formulation. A factorized family may miss dependencies or alternative modes; imperfect optimization adds another approximation error.
• MCMC constructs a Markov chain whose stationary distribution is the target posterior under appropriate conditions. Finite runs may mix slowly, retain strong sample correlation, or miss modes. Convergence and effective sample size matter; merely collecting many steps does not establish an accurate posterior approximation.
• Monte Carlo dropout keeps dropout on at test time and treats the varying outputs as samples. Gal and Ghahramani's 2016 paper casts dropout training as “approximate Bayesian inference in deep Gaussian processes”, under the paper’s assumptions. Turning on arbitrary dropout at inference does not by itself establish that the output spread is a calibrated posterior uncertainty estimate.

The prior, likelihood and observed data define the target posterior. A variational family restricts the approximation, not that target. Optimization error, family restrictions and finite-sample Monte Carlo error are different sources of uncertainty in the computation. Report which method and assumptions produced the intervals.

One comparison is worth knowing before committing to the machinery. Lakshminarayanan, Pritzel and Blundell proposed deep ensembles as a non-Bayesian alternative and reported uncertainty estimates “as good or better than approximate Bayesian” methods, while being far simpler to implement. For a deployment decision, compare the predictive quality, uncertainty behavior, and cost on the intended workload rather than treating that benchmark result as a universal ordering.

References

• Gal, Y., and Ghahramani, Z. (2016). Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. arXiv:1506.02142. arxiv.org
• Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2017). Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. arXiv:1612.01474. arxiv.org
• NumPy documentation, numpy.linalg.inv.
• NumPy documentation, numpy.linalg.lstsq.
• NumPy documentation, numpy.linalg.solve.
• Blei, D. M., Kucukelbir, A., and McAuliffe, J. D. (2017). Variational Inference: A Review for Statisticians.