Neural Network Representation: Understanding the Layers and Notation
A neural network diagram represents a sequence of computations that transforms input features into a prediction. To understand how a neural network works, it helps to identify its layers, activations, parameters, and dimensional structure.
This article focuses on a neural network with one hidden layer.
The Three Parts of the Network
Consider a neural network with three input features, four hidden units, and one output unit. Its structure consists of:
- An input layer
- A hidden layer
- An output layer
Each part has a different role in transforming the input into the final prediction.
The Input Layer
The input layer contains the features supplied to the neural network:\[ x= \begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix} \]
In this example, the network receives three input features, so:\[ x\in\mathbb{R}^{3\times1} \]
The input layer does not learn new features or perform the same kind of computation as the later layers. It simply passes the input values to the first hidden layer.
The Hidden Layer
The next group of nodes is called the hidden layer. In this example, it contains four hidden units.
The word “hidden” means that the correct values of these units are not directly observed in the labeled data.
In supervised learning, the available data contains:
- Input features \(x\)
- Target output \(y\)
However, it does not provide target values for the hidden units. The network must learn useful intermediate representations by adjusting its parameters during training.
The Output Layer
The final layer is the output layer. In this example, it contains a single unit responsible for generating the prediction:\[ \hat{y} \]
For binary classification, this output is commonly interpreted as the estimated probability that the label equals 1:\[ \hat{y}=P(y=1\mid x) \]
The output layer receives the hidden layer’s activations and transforms them into the final result.
What Are Activations?
The values passed from one layer to the next are called activations.
The input itself can be treated as the activations of layer zero:\[ a^{[0]}=x \]
The hidden layer generates:\[ a^{[1]} \]
The output layer generates:\[ a^{[2]} \]
Since the final activation is the network’s prediction:\[ a^{[2]}=\hat{y} \]
The superscript in square brackets identifies the layer associated with a quantity.
Hidden-Layer Activations
Because the hidden layer contains four units, each unit produces one activation:\[ a_1^{[1]},\quad a_2^{[1]},\quad a_3^{[1]},\quad a_4^{[1]} \]
These values are stacked vertically to form the activation vector:\[ a^{[1]} = \begin{bmatrix} a_1^{[1]}\\ a_2^{[1]}\\ a_3^{[1]}\\ a_4^{[1]} \end{bmatrix} \]
Therefore:\[ a^{[1]}\in\mathbb{R}^{4\times1} \]
The vector has four elements because the hidden layer has four units.
Understanding the Indices
A quantity such as:\[ a_i^{[l]} \]
contains two pieces of information:
- The superscript \([l]\) identifies the layer.
- The subscript \(i\) identifies a particular unit in that layer.
For example:\[ a_2^{[1]} \]
means the activation of the second unit in the first layer.
Square brackets are used for layer numbers. Parentheses are generally used to identify individual data examples:\[ x^{(i)} \]
Thus:\[ a_2^{[1](i)} \]
would refer to the activation of the second unit in layer one for example \(i\).
Why It Is Called a Two-Layer Neural Network
The diagram contains:
- An input layer
- One hidden layer
- One output layer
It may appear to contain three layers, but neural-network convention does not count the input layer when naming the network’s depth.
Therefore:
- The input layer is designated layer 0.
- The hidden layer is layer 1.
- The output layer is layer 2.
This network is called a two-layer neural network because only the hidden and output layers are counted.
It can also be described as a neural network with one hidden layer.
Parameters of the Hidden Layer
The hidden layer has two parameter sets:\[ W^{[1]} \]
and:\[ b^{[1]} \]
The weight matrix \(W^{[1]}\) connects the three input features to the four hidden units.
Its dimensions are:\[ W^{[1]}\in\mathbb{R}^{4\times3} \]
The four rows correspond to the four hidden units, while the three columns correspond to the three input features.
The bias vector contains one bias for each hidden unit:\[ b^{[1]}\in\mathbb{R}^{4\times1} \]
These parameters are used to compute:\[ z^{[1]}=W^{[1]}a^{[0]}+b^{[1]} \]
followed by an activation function:\[ a^{[1]}=g^{[1]}\left(z^{[1]}\right) \]
Parameters of the Output Layer
The output layer also has weights and biases:\[ W^{[2]} \]
and:\[ b^{[2]} \]
The output layer receives four hidden activations and produces one output. Therefore:\[ W^{[2]}\in\mathbb{R}^{1\times4} \]
The single row corresponds to the one output unit, while the four columns correspond to the four hidden units.
The output layer has one bias:\[ b^{[2]}\in\mathbb{R}^{1\times1} \]
The output calculation is:\[ z^{[2]}=W^{[2]}a^{[1]}+b^{[2]} \]\[ a^{[2]}=g^{[2]}\left(z^{[2]}\right) \]
and:\[ \hat{y}=a^{[2]} \]
Dimension Summary
| Quantity | Shape | Meaning |
|---|---|---|
| \(x=a^{[0]}\) | \((3,1)\) | Input features |
| \(W^{[1]}\) | \((4,3)\) | Weights from the input layer to the hidden layer |
| \(b^{[1]}\) | \((4,1)\) | Hidden-layer biases |
| \(z^{[1]}\) | \((4,1)\) | Hidden-layer linear values |
| \(a^{[1]}\) | \((4,1)\) | Hidden-layer activations |
| \(W^{[2]}\) | \((1,4)\) | Weights from the hidden layer to the output |
| \(b^{[2]}\) | \((1,1)\) | Output-layer bias |
| \(z^{[2]}\) | \((1,1)\) | Output-layer linear value |
| \(a^{[2]}=\hat{y}\) | \((1,1)\) | Final prediction |
General Rule for Parameter Dimensions
For a layer \(l\), let:\[ n^{[l]} \]
represent the number of units in that layer.
The general dimensions are:\[ W^{[l]} \in \mathbb{R}^{n^{[l]}\times n^{[l-1]}} \]
and:\[ b^{[l]} \in \mathbb{R}^{n^{[l]}\times1} \]
In this example:\[ n^{[0]}=3,\qquad n^{[1]}=4,\qquad n^{[2]}=1 \]
Therefore:\[ W^{[1]}:(4,3) \]\[ b^{[1]}:(4,1) \]\[ W^{[2]}:(1,4) \]\[ b^{[2]}:(1,1) \]
The Complete Flow of Information
The network transforms the input through the following sequence:\[ a^{[0]}=x \]\[ z^{[1]}=W^{[1]}a^{[0]}+b^{[1]} \]\[ a^{[1]}=g^{[1]}\left(z^{[1]}\right) \]\[ z^{[2]}=W^{[2]}a^{[1]}+b^{[2]} \]\[ a^{[2]}=g^{[2]}\left(z^{[2]}\right) \]\[ \hat{y}=a^{[2]} \]
The input layer passes the features to the hidden layer. The hidden layer learns an intermediate representation, and the output layer converts that representation into a prediction.
Key Takeaway
A neural network with one hidden layer contains an input layer, a hidden layer, and an output layer. By convention, the input layer is not counted, so the model is called a two-layer neural network.
Its core notation is:\[ a^{[0]}=x \]\[ a^{[1]}=\text{hidden-layer activations} \]\[ a^{[2]}=\hat{y} \]
For a network with three inputs, four hidden units, and one output:\[ W^{[1]}:(4,3),\qquad b^{[1]}:(4,1) \]\[ W^{[2]}:(1,4),\qquad b^{[2]}:(1,1) \]
The hidden units are called hidden because their correct values are not provided directly in the labeled data. Instead, the network learns these intermediate representations while learning to map \(x\) to \(y\).