Neural Networks Overview
A neural network can be understood as a sequence of computations similar to logistic regression, repeated across multiple layers. Each layer performs a linear calculation followed by an activation function, and the final layer produces the model’s prediction.
The network first performs forward propagation to calculate its output and loss. It then performs backward propagation to calculate the derivatives needed to update its parameters.
Logistic Regression as the Starting Point
For a single input vector \(x\), logistic regression uses parameters \(w\) and \(b\) to compute:\[ z=w^Tx+b \]
It then applies the sigmoid activation function:\[ a=\sigma(z) \]
where:\[ \sigma(z)=\frac{1}{1+e^{-z}} \]
The activation \(a\) is also the model’s predicted output:\[ a=\hat{y} \]
The prediction and true label are then used to calculate the loss:\[ L(\hat{y},y) \]
For binary classification, this is commonly the binary cross-entropy loss:\[ L(\hat{y},y) = -\left[ y\log\hat{y} + (1-y)\log(1-\hat{y}) \right] \]
The overall sequence is:\[ x \rightarrow z \rightarrow a=\hat{y} \rightarrow L \]
From Logistic Regression to a Neural Network
A neural network is formed by stacking multiple computational units together. Each unit performs the same two basic operations:
- Compute a linear value \(z\).
- Apply an activation function to produce \(a\).
Instead of computing these operations only once, a neural network performs them across multiple layers.
For a network with one hidden layer:\[ x \rightarrow z^{[1]} \rightarrow a^{[1]} \rightarrow z^{[2]} \rightarrow a^{[2]}=\hat{y} \rightarrow L \]
The first pair of calculations belongs to the hidden layer, and the second pair belongs to the output layer.
Forward Propagation Through the Hidden Layer
The input features enter the first layer:\[ z^{[1]}=W^{[1]}x+b^{[1]} \]
The activation function is then applied:\[ a^{[1]}=g^{[1]}\left(z^{[1]}\right) \]
If the hidden layer uses the sigmoid function:\[ a^{[1]}=\sigma\left(z^{[1]}\right) \]
Unlike logistic regression, \(z^{[1]}\) and \(a^{[1]}\) are generally vectors because a hidden layer usually contains multiple units.
Each hidden unit has its own weights and bias, so the units can learn different intermediate features from the same input.
Forward Propagation Through the Output Layer
The hidden-layer activations become the input to the output layer:\[ z^{[2]}=W^{[2]}a^{[1]}+b^{[2]} \]
The output activation is:\[ a^{[2]}=g^{[2]}\left(z^{[2]}\right) \]
For binary classification, the output layer commonly uses the sigmoid function:\[ a^{[2]}=\sigma\left(z^{[2]}\right) \]
The final activation is the prediction:\[ \boxed{a^{[2]}=\hat{y}} \]
The complete forward propagation process is therefore:\[ z^{[1]}=W^{[1]}x+b^{[1]} \]\[ a^{[1]}=g^{[1]}\left(z^{[1]}\right) \]\[ z^{[2]}=W^{[2]}a^{[1]}+b^{[2]} \]\[ a^{[2]}=g^{[2]}\left(z^{[2]}\right)=\hat{y} \]
Understanding the Layer Notation
Neural-network notation uses superscripts in square brackets to identify layers:\[ z^{[1]},\quad a^{[1]},\quad W^{[1]},\quad b^{[1]} \]
belong to layer 1, while:\[ z^{[2]},\quad a^{[2]},\quad W^{[2]},\quad b^{[2]} \]
belong to layer 2.
The square brackets are important because parentheses are used for a different purpose.
Square brackets identify layers
\[ a^{[1]} \]
means the activations of layer 1.
Parentheses identify data examples
\[ x^{(i)} \]
means the input for example \(i\).
The combined notation:\[ a^{[1](i)} \]
means the activation of layer 1 for example \(i\).
Forward Propagation and Loss
Once the final prediction has been computed, the network evaluates how closely it matches the true label:\[ L\left(a^{[2]},y\right) \]
Because:\[ a^{[2]}=\hat{y} \]
this can also be written as:\[ L(\hat{y},y) \]
Forward propagation moves from left to right through the network:\[ x \rightarrow z^{[1]} \rightarrow a^{[1]} \rightarrow z^{[2]} \rightarrow a^{[2]} \rightarrow L \]
Its purpose is to calculate the prediction and the resulting loss.
Backward Propagation
After forward propagation calculates the loss, backward propagation computes how the loss changes with respect to the network’s intermediate values and parameters.
The calculation proceeds from right to left:\[ L \rightarrow da^{[2]} \rightarrow dz^{[2]} \rightarrow dW^{[2]},db^{[2]} \rightarrow da^{[1]} \rightarrow dz^{[1]} \rightarrow dW^{[1]},db^{[1]} \]
These derivatives describe how the parameters should change to reduce the loss.
For example:\[ dW^{[2]} = \frac{\partial L}{\partial W^{[2]}} \]
and:\[ db^{[2]} = \frac{\partial L}{\partial b^{[2]}} \]
Similar derivatives are calculated for the hidden layer:\[ dW^{[1]} = \frac{\partial L}{\partial W^{[1]}} \]\[ db^{[1]} = \frac{\partial L}{\partial b^{[1]}} \]
Updating the Parameters
After the derivatives have been calculated, gradient descent updates the weights and biases.
For layer 1:\[ W^{[1]} := W^{[1]}-\alpha dW^{[1]} \]\[ b^{[1]} := b^{[1]}-\alpha db^{[1]} \]
For layer 2:\[ W^{[2]} := W^{[2]}-\alpha dW^{[2]} \]\[ b^{[2]} := b^{[2]}-\alpha db^{[2]} \]
Here, \(\alpha\) is the learning rate. These updates are repeated so that the network gradually learns parameter values that produce a lower loss.
The Complete Training Cycle
One training iteration consists of three main stages.
1. Forward propagation
Compute the hidden activations and prediction:\[ x \rightarrow z^{[1]} \rightarrow a^{[1]} \rightarrow z^{[2]} \rightarrow a^{[2]}=\hat{y} \]
2. Backward propagation
Compute the derivatives:\[ dW^{[2]},\ db^{[2]},\ dW^{[1]},\ db^{[1]} \]
3. Parameter updates
Use gradient descent to update:\[ W^{[1]},\ b^{[1]},\ W^{[2]},\ b^{[2]} \]
The network repeats this cycle until its parameters converge or the desired training condition is reached.
Neural Networks as Repeated Logistic Regression
The central intuition is that logistic regression performs one linear calculation followed by one activation:\[ z\rightarrow a \]
A neural network repeats this structure:\[ z^{[1]} \rightarrow a^{[1]} \rightarrow z^{[2]} \rightarrow a^{[2]} \]
This resemblance is useful for understanding the mechanics, although a neural network is more expressive because its hidden layer learns intermediate representations before generating the final output.
The same pattern extends to deeper networks:\[ z^{[1]}\rightarrow a^{[1]} \rightarrow z^{[2]}\rightarrow a^{[2]} \rightarrow \cdots \rightarrow z^{[L]}\rightarrow a^{[L]} \]
Key Takeaway
A neural network with one hidden layer performs four main forward calculations:\[ z^{[1]}=W^{[1]}x+b^{[1]} \]\[ a^{[1]}=g^{[1]}\left(z^{[1]}\right) \]\[ z^{[2]}=W^{[2]}a^{[1]}+b^{[2]} \]\[ a^{[2]}=g^{[2]}\left(z^{[2]}\right)=\hat{y} \]
Forward propagation calculates the prediction and loss. Backward propagation moves in the opposite direction to calculate the derivatives needed for gradient descent.
The simplest way to understand this structure is as a repeated pattern:\[ \boxed{\text{linear calculation}\rightarrow\text{activation}} \]
Each additional layer repeats that pattern, allowing the network to learn increasingly complex relationships between its inputs and outputs.