Neural Networks

A neural network is a parameterized computation that transforms inputs into outputs through connected operations. A common feedforward network composes layers of weighted sums and nonlinear functions. Training adjusts its weights and biases using examples and an objective. The same learned parameters are then used to make predictions on new inputs. This mathematical construction is more useful for understanding modern networks than treating them as literal models of a brain.

What a layer computes

In a dense layer, each output unit combines the input values: z = Wx + b. The matrix W says how strongly each input contributes to each output; the bias vector b shifts those outputs. An activation function then computes h = φ(z). A layer with three inputs and four outputs has 12 weights and four biases. A following layer consumes those four output values as its input representation.

For a single illustrative unit, x = [2, −1], w = [0.5, 1], and b = 0.2 give z = 0.5 × 2 + 1 × (−1) + 0.2 = 0.2. ReLU returns max(0, z), so this unit outputs 0.2. With x = [0, −1], the same unit gives z = −0.8 and outputs zero. Its response depends on both the input and the learned parameters. Many such units form a representation that the next layer combines; an individual unit need not correspond to a human-named concept.

Why nonlinearities matter

Two purely affine layers collapse into one affine map: W₂(W₁x + b₁) + b₂ = (W₂W₁)x + (W₂b₁ + b₂). Extra affine layers can alter parameterization and optimization, but do not by themselves introduce a nonlinear input-output function. Intervening activation functions can change that expressive capacity. ReLU, despite its simplicity, switches which linear region applies as inputs cross unit boundaries.

For binary inputs, consider XOR: output 1 when exactly one of x₁ and x₂ is 1. One affine threshold cannot separate the two positive corners from the two negative corners. But let h₁ = ReLU(x₁ + x₂) and h₂ = ReLU(x₁ + x₂ − 1), and output h₁ − 2h₂. Input sums 0, 1, and 2 then produce 0, 1, and 0. This is a hand-constructed example of nonlinear representation, not evidence that a training algorithm will always discover these weights.

How the parameters learn

A training step first performs the forward computation and compares its outputs with targets using a loss function. The loss specifies which errors matter. Squared error favors the conditional mean when minimizing the unrestricted population risk with finite second moments; absolute error favors a conditional median. A finite network fitted to finite data is constrained and need not attain either ideal target.

Backpropagation applies the chain rule through the computation graph to obtain derivatives of the loss with respect to parameters. The optimizer then uses those derivatives to update the parameters. These are separate jobs: a correct gradient does not specify the learning rate or guarantee a useful optimization path. For an illustrative scalar parameter with gradient 0.4 and learning rate 0.1, a plain gradient-descent update subtracts 0.04 from its current value.

Training repeats on batches, usually over multiple passes through the data. Initialization, gradient scale, learning rate, and regularization affect the result. A learning rate that is too large can cause unstable updates; one that is too small can make progress slow. Validation on data excluded from fitting helps distinguish better training fit from improvement on the task. Continued training can produce overfitting, especially when the model learns sample-specific noise.

Matching outputs and architecture to the task

For binary classification, a sigmoid transforms a score into a value between zero and one. A softmax normalizes several scores into class probabilities summing to one. A regression output can remain unconstrained when the target can be any real value. Match these choices to the loss implementation: a loss that already applies the transformation internally expects logits, not probabilities transformed a second time. A probability-shaped output is not automatically calibrated.

A dense network connects every input to every unit in the next layer. A convolutional network shares filters across locations, using local structure that is useful for images. A recurrent network updates a state across sequence positions. Attention uses input-dependent relationships to combine information across positions. These mechanisms change what interactions are easy to represent and how computation scales; they are not merely different layer counts or constraints on one fixed linear map.

Expressive power is not a learning guarantee

Universal-approximation results establish that suitable networks can approximate broad classes of functions under stated activation and domain conditions. For one-dimensional continuous functions on a compact interval, a concrete ReLU construction uses a fine piecewise-linear interpolant: shifted ReLU units introduce changes of slope, combined with an affine term. Finer intervals can improve approximation when the function is continuous. The classic results by Cybenko and Hornik provide related general foundations.

Existence does not tell us how much data is needed, whether the optimizer finds useful parameters, or how the model behaves outside the observed domain. Increasing width can improve fit while increasing memory and computation; its effect on generalization must be evaluated. Choosing a network therefore requires an input representation, a task-appropriate loss, a feasible training procedure, and evidence on new data. Depth alone does not settle any of those decisions.

References

• Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals, and Systems, 2, 303–314. doi.org
• Hornik, K. (1991). Approximation capabilities of multilayer feedforward networks. Neural Networks, 4, 251–257. doi.org
• NumPy documentation, numpy.linalg.lstsq. numpy.org