Building One Convolutional Layer in a Neural Network
A convolutional layer combines four operations:
- Convolve the input with multiple filters.
- Add one bias value for each filter.
- Apply a nonlinear activation function.
- Stack the resulting feature maps into an output volume.
This is the convolutional equivalent of a standard neural-network layer:\[ Z^{(l)} = W^{(l)}A^{(l-1)}+b^{(l)} \]\[ A^{(l)} = g\left(Z^{(l)}\right) \]
The difference is that a convolutional layer uses local connectivity and parameter sharing instead of a fully connected matrix multiplication.
From an Input Volume to an Output Volume
Consider an RGB input volume:\[ A^{(0)} = X \in \mathbb{R}^{6\times6\times3} \]
Suppose the convolutional layer contains two filters. Each filter has shape:\[ 3\times3\times3 \]
The final dimension is 3 because every standard convolutional filter must span all three input channels.
With stride 1 and no padding, each filter produces a \(4\times4\) feature map:\[ 6-3+1=4 \]
Therefore:\[ Z^{(1,1)} \in \mathbb{R}^{4\times4} \]
and:\[ Z^{(1,2)} \in \mathbb{R}^{4\times4} \]
The second superscript identifies which filter produced the map.
Add One Bias per Filter
Each filter has one associated bias value.
For the first filter:\[ b^{(1,1)}\in\mathbb{R} \]
For the second filter:\[ b^{(1,2)}\in\mathbb{R} \]
The first bias is added to every position in the first feature map:\[ \widetilde{Z}^{(1,1)}_{i,j} = Z^{(1,1)}_{i,j} + b^{(1,1)} \]
Similarly:\[ \widetilde{Z}^{(1,2)}_{i,j} = Z^{(1,2)}_{i,j} + b^{(1,2)} \]
Although each bias is a single scalar, broadcasting applies it across the complete spatial feature map.
For example, if:\[ b^{(1,1)}=2 \]
then 2 is added to all 16 entries of the corresponding \(4\times4\) map.
Apply a Nonlinear Activation
After adding the bias, apply an activation function.
Using ReLU:\[ g(z)=\max(0,z) \]
the activation map from filter \(k\) is:\[ A^{(1,k)} = \operatorname{ReLU} \left( Z^{(1,k)}+b^{(1,k)} \right) \]
For the two filters:\[ A^{(1,1)} \in \mathbb{R}^{4\times4} \]\[ A^{(1,2)} \in \mathbb{R}^{4\times4} \]
ReLU is applied element by element.
Without a nonlinear activation, stacking convolutional layers would still represent a linear transformation. The activation allows the network to learn nonlinear visual relationships.
Stack the Feature Maps
The two activated feature maps are stacked along the channel dimension:\[ A^{(1)} = \operatorname{stack} \left( A^{(1,1)}, A^{(1,2)} \right) \]
The output volume has shape:\[ A^{(1)} \in \mathbb{R}^{4\times4\times2} \]
The complete transformation is therefore:\[ 6\times6\times3 \longrightarrow 4\times4\times2 \]
The output has two channels because the layer contains two filters.
The number of output channels equals the number of filters in the convolutional layer.
Correcting the Output-Dimension Description
With two filters, the correct output shape is:\[ 4\times4\times2 \]
It is not \(4\times4\times4\). Each of the two filters produces one \(4\times4\) feature map, and stacking those maps creates two output channels.
Correspondence with a Fully Connected Layer
A standard dense layer computes:\[ Z^{(l)} = W^{(l)}A^{(l-1)} + b^{(l)} \]\[ A^{(l)} = g\left(Z^{(l)}\right) \]
A convolutional layer follows the same pattern:\[ Z^{(l)} = W^{(l)} * A^{(l-1)} + b^{(l)} \]\[ A^{(l)} = g\left(Z^{(l)}\right) \]
Here, the symbol \(*\) denotes the operation commonly called convolution in deep learning.
The correspondence is:
| Dense layer | Convolutional layer |
|---|---|
| Weight matrix | Collection of filters |
| Matrix multiplication | Convolution |
| Bias per output unit | Bias per output channel |
| Activation function | Elementwise activation |
| Output vector | Output volume |
The Linear Part of a Convolutional Layer
For output channel \(k\), the pre-activation at spatial position \((i,j)\) is:\[ Z^{(l)}_{i,j,k} = \sum_{a=0}^{f_H-1} \sum_{b=0}^{f_W-1} \sum_{c=0}^{n_C^{(l-1)}-1} W^{(l)}_{a,b,c,k} A^{(l-1)}_{i+a,j+b,c} + b^{(l)}_k \]
This assumes:
- Stride 1
- No padding
- A channels-last representation
The activation is:\[ A^{(l)}_{i,j,k} = g\left( Z^{(l)}_{i,j,k} \right) \]
Each output value is therefore created by:
- Selecting a local region from the previous layer
- Multiplying it by filter \(k\)
- Summing across height, width, and input channels
- Adding the bias for filter \(k\)
- Applying the activation function
General Form with Stride and Padding
Let layer \(l\) use:
- Filter height \(f_H^{(l)}\)
- Filter width \(f_W^{(l)}\)
- Vertical padding \(p_H^{(l)}\)
- Horizontal padding \(p_W^{(l)}\)
- Vertical stride \(s_H^{(l)}\)
- Horizontal stride \(s_W^{(l)}\)
Then:\[ Z^{(l)}_{i,j,k} = \sum_{a=0}^{f_H^{(l)}-1} \sum_{b=0}^{f_W^{(l)}-1} \sum_{c=0}^{n_C^{(l-1)}-1} W^{(l)}_{a,b,c,k} A_{\text{pad}}^{(l-1)} \left( i s_H^{(l)}+a, j s_W^{(l)}+b, c \right) + b_k^{(l)} \]
The output activation remains:\[ A^{(l)}=g\left(Z^{(l)}\right) \]
Increasing the Number of Filters
Suppose the same layer uses 10 filters instead of two.
Each filter produces one \(4\times4\) feature map:\[ A^{(1,1)}, A^{(1,2)}, \ldots, A^{(1,10)} \]
Stacking them gives:\[ A^{(1)} \in \mathbb{R}^{4\times4\times10} \]
Therefore:\[ 6\times6\times3 \overset{10\text{ filters}}{\longrightarrow} 4\times4\times10 \]
The filters may learn to detect different patterns, such as:
- Vertical edges
- Horizontal edges
- Diagonal edges
- Corners
- Color transitions
- Textures
- Curves
- Other task-specific features
Counting the Parameters
Suppose the layer contains 10 filters, each with shape:\[ 3\times3\times3 \]
Each filter contains:\[ 3\cdot3\cdot3=27 \]
weights.
Each filter also has one bias:\[ 27+1=28 \]
parameters per filter.
With 10 filters:\[ 10\cdot28=280 \]
Therefore, the layer has:\[ \boxed{280\text{ parameters}} \]
This consists of:
- 270 filter weights
- 10 bias values
Parameter Count Is Independent of Image Size
The same 10 filters can be applied to:
- A \(32\times32\) image
- A \(1000\times1000\) image
- A \(5000\times5000\) image
The number of trainable parameters remains:\[ 280 \]
Larger images produce larger feature maps and require more computation, but they do not require additional filter parameters.
This is possible because the same filter values are reused across every spatial position.
Why This Reduces Overfitting
A fully connected layer uses separate weights for many input-output relationships. Its parameter count can grow dramatically as image resolution increases.
A convolutional layer uses:
- Local connectivity
- Parameter sharing
A feature detector learned in one part of the image can be applied everywhere else.
This creates a strong inductive bias:
A useful visual pattern may appear at different spatial positions.
The smaller parameter count often reduces the risk of overfitting compared with a dense layer applied directly to a high-resolution image.
It does not eliminate overfitting, but it makes visual learning substantially more data-efficient.
General Notation for Layer \(l\)
Let the input to convolutional layer \(l\) be:\[ A^{(l-1)} \in \mathbb{R}^{ n_H^{(l-1)} \times n_W^{(l-1)} \times n_C^{(l-1)} } \]
The dimensions represent:
- \(n_H^{(l-1)}\): input height
- \(n_W^{(l-1)}\): input width
- \(n_C^{(l-1)}\): number of input channels
The layer produces:\[ A^{(l)} \in \mathbb{R}^{ n_H^{(l)} \times n_W^{(l)} \times n_C^{(l)} } \]
where:
- \(n_H^{(l)}\) is the output height.
- \(n_W^{(l)}\) is the output width.
- \(n_C^{(l)}\) is the number of output channels.
Convolutional-Layer Hyperparameters
A convolutional layer is defined by several important values.
| Symbol | Meaning |
|---|---|
| \(f_H^{(l)},f_W^{(l)}\) | Spatial filter dimensions |
| \(p_H^{(l)},p_W^{(l)}\) | Padding |
| \(s_H^{(l)},s_W^{(l)}\) | Stride |
| \(n_C^{(l)}\) | Number of filters and output channels |
| \(g^{(l)}\) | Activation function |
For square filters and equal spatial settings, these may be shortened to:\[ f^{(l)},\qquad p^{(l)},\qquad s^{(l)} \]
Output Height and Width
The output height is:\[ n_H^{(l)} = \left\lfloor \frac{ n_H^{(l-1)} + 2p_H^{(l)} – f_H^{(l)} }{ s_H^{(l)} } \right\rfloor +1 \]
The output width is:\[ n_W^{(l)} = \left\lfloor \frac{ n_W^{(l-1)} + 2p_W^{(l)} – f_W^{(l)} }{ s_W^{(l)} } \right\rfloor +1 \]
For square inputs and filters:\[ n^{(l)} = \left\lfloor \frac{ n^{(l-1)} + 2p^{(l)} – f^{(l)} }{ s^{(l)} } \right\rfloor +1 \]
Number of Output Channels
The output channel count is:\[ n_C^{(l)} = \text{number of filters in layer }l \]
For example:
- Two filters produce two channels.
- Ten filters produce ten channels.
- Sixty-four filters produce sixty-four channels.
This quantity is selected as an architectural hyperparameter.
Shape of One Filter
Each standard convolutional filter must span every input channel:\[ f_H^{(l)} \times f_W^{(l)} \times n_C^{(l-1)} \]
For square filters:\[ f^{(l)} \times f^{(l)} \times n_C^{(l-1)} \]
If the previous layer contains 64 channels, every standard filter in the current layer has depth 64.
Shape of the Complete Weight Tensor
Because layer \(l\) contains \(n_C^{(l)}\) filters, the complete weight tensor has shape:\[ W^{(l)} \in \mathbb{R}^{ f_H^{(l)} \times f_W^{(l)} \times n_C^{(l-1)} \times n_C^{(l)} } \]
Under a square-filter convention:\[ W^{(l)} \in \mathbb{R}^{ f^{(l)} \times f^{(l)} \times n_C^{(l-1)} \times n_C^{(l)} } \]
The final dimension indexes the filters.
Shape of the Bias Tensor
There is one bias for every output filter:\[ b^{(l)} \in \mathbb{R}^{n_C^{(l)}} \]
In a channels-last implementation, the bias may be represented in a broadcast-friendly form:\[ b^{(l)} \in \mathbb{R}^{1\times1\times1\times n_C^{(l)}} \]
The leading dimensions allow the bias to be broadcast across:
- All examples
- Every output row
- Every output column
Only the output-channel dimension varies.
General Parameter Count
The number of filter weights is:\[ f_H^{(l)} f_W^{(l)} n_C^{(l-1)} n_C^{(l)} \]
The number of biases is:\[ n_C^{(l)} \]
Therefore, the total number of trainable parameters is:\[ n_C^{(l)} \left( f_H^{(l)} f_W^{(l)} n_C^{(l-1)} +1 \right) \]
For square filters:\[ n_C^{(l)} \left( \left(f^{(l)}\right)^2 n_C^{(l-1)} +1 \right) \]
Notice that neither input height nor input width appears in this expression.
Batch Dimensions
For one channels-last example:\[ A^{(l)} \in \mathbb{R}^{ n_H^{(l)} \times n_W^{(l)} \times n_C^{(l)} } \]
For a mini-batch of \(m\) examples:\[ A^{(l)} \in \mathbb{R}^{ m \times n_H^{(l)} \times n_W^{(l)} \times n_C^{(l)} } \]
This layout is commonly abbreviated as:\[ \text{NHWC} \]
where:
- N is batch size.
- H is height.
- W is width.
- C is channels.
Channels-First Layout
Some systems instead use:\[ A^{(l)} \in \mathbb{R}^{ m \times n_C^{(l)} \times n_H^{(l)} \times n_W^{(l)} } \]
This is called:\[ \text{NCHW} \]
Neither layout changes the underlying mathematics. The important requirement is consistency.
| Layout | Batch tensor order |
|---|---|
| NHWC | Batch, height, width, channels |
| NCHW | Batch, channels, height, width |
When reading or writing convolutional code, always verify which convention is being used.
Complete Layer Summary
A convolutional layer receives:\[ A^{(l-1)} \]
and computes:\[ Z^{(l)} = \operatorname{Conv} \left( A^{(l-1)},W^{(l)} \right) + b^{(l)} \]
It then applies an activation:\[ A^{(l)} = g^{(l)} \left( Z^{(l)} \right) \]
The output dimensions are:\[ n_H^{(l)} \times n_W^{(l)} \times n_C^{(l)} \]
where:\[ n_H^{(l)} = \left\lfloor \frac{ n_H^{(l-1)} + 2p_H^{(l)} – f_H^{(l)} }{ s_H^{(l)} } \right\rfloor +1 \]\[ n_W^{(l)} = \left\lfloor \frac{ n_W^{(l-1)} + 2p_W^{(l)} – f_W^{(l)} }{ s_W^{(l)} } \right\rfloor +1 \]
and:\[ n_C^{(l)} = \text{number of filters} \]
Key Takeaway
One convolutional layer performs:\[ \text{convolution} \rightarrow \text{bias addition} \rightarrow \text{nonlinear activation} \rightarrow \text{channel stacking} \]
For a layer with input:\[ 6\times6\times3 \]
and two \(3\times3\times3\) filters, using stride 1 and no padding, the output is:\[ 4\times4\times2 \]
Each filter produces one output channel and has one associated bias.
The complete parameter count for a standard convolutional layer is:\[ n_C^{(l)} \left( f_H^{(l)} f_W^{(l)} n_C^{(l-1)} +1 \right) \]
Because convolutional filters are shared across spatial positions, this parameter count does not depend on the image height or width. This efficient parameter sharing is one of the main reasons convolutional networks work so well with large images.