A LeNet-Inspired Convolutional Neural Network

A complete convolutional neural network commonly combines three building blocks:

• Convolutional layers
• Pooling layers
• Fully connected layers

A traditional image-classification architecture often follows this pattern:\[ \text{INPUT} \rightarrow \text{CONV} \rightarrow \text{POOL} \rightarrow \text{CONV} \rightarrow \text{POOL} \rightarrow \text{FC} \rightarrow \text{FC} \rightarrow \text{SOFTMAX} \]

This article develops a network inspired by LeNet-5, an influential early convolutional architecture created by Yann LeCun and collaborators.

The example is not an exact reproduction of the original LeNet-5. It uses a similar structure to demonstrate how convolutional, pooling, and fully connected layers work together.

Classification Task

Suppose the input is an RGB image with dimensions:\[ 32\times32\times3 \]

The network must recognize one of 10 digit classes:\[ 0,1,2,\ldots,9 \]

The output is a probability distribution:\[ \hat{y} \in \mathbb{R}^{10} \]

where:\[ \sum_{k=0}^{9}\hat{y}_k=1 \]

The predicted class is:\[ \hat{k} = \arg\max_k\hat{y}_k \]

Complete Architecture

The network has the following structure:\[ 32\times32\times3 \]\[ \downarrow\quad \text{CONV 1: }5\times5,\ 6\text{ filters} \]\[ 28\times28\times6 \]\[ \downarrow\quad \text{MAX POOL 1: }2\times2,\ s=2 \]\[ 14\times14\times6 \]\[ \downarrow\quad \text{CONV 2: }5\times5,\ 16\text{ filters} \]\[ 10\times10\times16 \]\[ \downarrow\quad \text{MAX POOL 2: }2\times2,\ s=2 \]\[ 5\times5\times16 \]\[ \downarrow\quad \text{FLATTEN} \]\[ 400 \]\[ \downarrow\quad \text{FC 3} \]\[ 120 \]\[ \downarrow\quad \text{FC 4} \]\[ 84 \]\[ \downarrow\quad \text{SOFTMAX} \]\[ 10 \]

Correcting the Architecture Description

The intended second convolution uses 16 filters, not 10.

The correct transformation is:\[ 14\times14\times6 \overset{ 5\times5,\ 16\text{ filters} }{ \longrightarrow } 10\times10\times16 \]

followed by:\[ 10\times10\times16 \overset{ 2\times2\text{ max pooling} }{ \longrightarrow } 5\times5\times16 \]

The complete spatial progression is:\[ 32 \rightarrow 28 \rightarrow 14 \rightarrow 10 \rightarrow 5 \]

Input Layer

The input activation is:\[ A^{(0)} = X \in \mathbb{R}^{32\times32\times3} \]

The total number of input values is:\[ 32\cdot32\cdot3 = 3{,}072 \]

The input layer itself has no trainable parameters.

Convolutional Layer 1

The first convolutional layer uses:

• Filter size: \(5\times5\)
• Stride: 1
• Padding: 0
• Number of filters: 6

Therefore:\[ f^{(1)}=5 \]\[ s^{(1)}=1 \]\[ p^{(1)}=0 \]\[ n_C^{(1)}=6 \]

CONV 1 Output Dimensions

The output height is:\[ n_H^{(1)} = \left\lfloor \frac{ 32+2(0)-5 }{ 1 } \right\rfloor+1 \]\[ =27+1=28 \]

Similarly:\[ n_W^{(1)}=28 \]

Because the layer uses six filters:\[ n_C^{(1)}=6 \]

Thus:\[ A_{\text{conv1}} \in \mathbb{R}^{28\times28\times6} \]

The transformation is:\[ 32\times32\times3 \longrightarrow 28\times28\times6 \]

CONV 1 Parameters

Each filter has shape:\[ 5\times5\times3 \]

The number of weights per filter is:\[ 5\cdot5\cdot3=75 \]

Including one bias:\[ 75+1=76 \]

With six filters:\[ \text{CONV 1 parameters} = 6(76) = 456 \]

CONV 1 Activation Count

The output contains:\[ 28\cdot28\cdot6 = 4{,}704 \]

activations per example.

Max-Pooling Layer 1

The first pooling layer uses:

• Window size: \(2\times2\)
• Stride: 2
• Padding: 0
• Operation: maximum

Therefore:\[ f_{\text{pool}}^{(1)}=2 \]\[ s_{\text{pool}}^{(1)}=2 \]

The output height is:\[ \left\lfloor \frac{28-2}{2} \right\rfloor+1 = 14 \]

The output width is also 14.

Pooling operates independently on every channel, so the channel count remains 6:\[ A_{\text{pool1}} \in \mathbb{R}^{14\times14\times6} \]

The transformation is:\[ 28\times28\times6 \longrightarrow 14\times14\times6 \]

POOL 1 Parameters and Activations

Max pooling has no trainable parameters:\[ \text{POOL 1 parameters}=0 \]

The output contains:\[ 14\cdot14\cdot6 = 1{,}176 \]

activations.

Convolutional Layer 2

The second convolutional layer receives:\[ A_{\text{pool1}} \in \mathbb{R}^{14\times14\times6} \]

It uses:

• Filter size: \(5\times5\)
• Stride: 1
• Padding: 0
• Number of filters: 16

Therefore:\[ f^{(2)}=5 \]\[ s^{(2)}=1 \]\[ p^{(2)}=0 \]\[ n_C^{(2)}=16 \]

CONV 2 Output Dimensions

The output height is:\[ n_H^{(2)} = \left\lfloor \frac{ 14+2(0)-5 }{ 1 } \right\rfloor+1 \]\[ =9+1=10 \]

Similarly:\[ n_W^{(2)}=10 \]

Because there are 16 filters:\[ n_C^{(2)}=16 \]

Therefore:\[ A_{\text{conv2}} \in \mathbb{R}^{10\times10\times16} \]

The transformation is:\[ 14\times14\times6 \longrightarrow 10\times10\times16 \]

CONV 2 Filter Dimensions

Each filter must span all six input channels:\[ 5\times5\times6 \]

The complete weight tensor has shape:\[ W^{(2)} \in \mathbb{R}^{5\times5\times6\times16} \]

The bias contains one value per output filter:\[ b^{(2)} \in \mathbb{R}^{16} \]

CONV 2 Parameter Count

Each filter contains:\[ 5\cdot5\cdot6=150 \]

weights and one bias:\[ 150+1=151 \]

With 16 filters:\[ \text{CONV 2 parameters} = 16(151) \]\[ =2{,}416 \]

CONV 2 Activation Count

The output contains:\[ 10\cdot10\cdot16 = 1{,}600 \]

activations.

Max-Pooling Layer 2

The second max-pooling layer uses:\[ f=2, \qquad s=2, \qquad p=0 \]

The output height is:\[ \left\lfloor \frac{10-2}{2} \right\rfloor+1 = 5 \]

The output width is also 5.

The number of channels remains 16:\[ A_{\text{pool2}} \in \mathbb{R}^{5\times5\times16} \]

Thus:\[ 10\times10\times16 \longrightarrow 5\times5\times16 \]

POOL 2 Parameters and Activations

The pooling layer contains no trainable parameters:\[ \text{POOL 2 parameters}=0 \]

Its output contains:\[ 5\cdot5\cdot16 = 400 \]

activations.

Flattening the Feature Volume

The pooled feature volume is:\[ 5\times5\times16 \]

The total number of values is:\[ 5\cdot5\cdot16 = 400 \]

Flattening converts the volume into a column vector:\[ a_{\text{flat}} \in \mathbb{R}^{400} \]

The operation is:\[ 5\times5\times16 \longrightarrow 400 \]

Flattening changes only the representation’s shape. It has no trainable parameters.

Fully Connected Layer 3

The first dense layer maps 400 inputs to 120 outputs:\[ a_{\text{flat}} \in \mathbb{R}^{400} \]\[ A^{(3)} \in \mathbb{R}^{120} \]

The computation is:\[ Z^{(3)} = W^{(3)}a_{\text{flat}} + b^{(3)} \]\[ A^{(3)} = g\left(Z^{(3)}\right) \]

The weight matrix has shape:\[ W^{(3)} \in \mathbb{R}^{120\times400} \]

The bias vector has shape:\[ b^{(3)} \in \mathbb{R}^{120} \]

FC 3 Parameter Count

The weight matrix contains:\[ 120\cdot400 = 48{,}000 \]

parameters.

The bias contributes:\[ 120 \]

additional parameters.

Therefore:\[ \text{FC 3 parameters} = 48{,}000+120 = 48{,}120 \]

This is far more than either convolutional layer contains.

Fully Connected Layer 4

The next dense layer maps 120 units to 84:\[ A^{(3)} \in \mathbb{R}^{120} \]\[ A^{(4)} \in \mathbb{R}^{84} \]

The computation is:\[ Z^{(4)} = W^{(4)}A^{(3)} + b^{(4)} \]\[ A^{(4)} = g\left(Z^{(4)}\right) \]

The weight matrix has shape:\[ W^{(4)} \in \mathbb{R}^{84\times120} \]

The bias vector has shape:\[ b^{(4)} \in \mathbb{R}^{84} \]

FC 4 Parameter Count

The weights contribute:\[ 84\cdot120 = 10{,}080 \]

parameters.

The bias contributes:\[ 84 \]

Therefore:\[ \text{FC 4 parameters} = 10{,}080+84 = 10{,}164 \]

Softmax Output Layer

The final layer receives 84 values and produces 10 class scores:\[ Z^{(5)} = W^{(5)}A^{(4)} + b^{(5)} \]

where:\[ W^{(5)} \in \mathbb{R}^{10\times84} \]

and:\[ b^{(5)} \in \mathbb{R}^{10} \]

The softmax probabilities are:\[ \hat{y}_k = \frac{ e^{Z_k^{(5)}} }{ \sum_{j=0}^{9}e^{Z_j^{(5)}} } \]

The output vector has shape:\[ \hat{y} \in \mathbb{R}^{10} \]

Output-Layer Parameter Count

The weights contribute:\[ 10\cdot84 = 840 \]

parameters.

The biases contribute:\[ 10 \]

Therefore:\[ \text{output parameters} = 840+10 = 850 \]

Complete Architecture Table

Total Parameter Count

Add the parameters from all trainable layers:\[ 456 + 2{,}416 + 48{,}120 + 10{,}164 + 850 \]\[ = 62{,}006 \]

Therefore:\[ \boxed{ \text{total trainable parameters} = 62{,}006 } \]

Where Are Most of the Parameters?

The two convolutional layers contain:\[ 456+2{,}416 = 2{,}872 \]

parameters.

The dense and output layers contain:\[ 48{,}120+10{,}164+850 = 59{,}134 \]

parameters.

Therefore, most parameters are in the fully connected portion:\[ \frac{59{,}134}{62{,}006} \approx 95.4\% \]

This illustrates a common property of traditional CNN architectures:

Convolutional layers often contain relatively few parameters, while fully connected layers may contain most of the model’s weights.

Modern architectures frequently use global average pooling or smaller classification heads to reduce this concentration of parameters.

Activation-Size Progression

The activation counts are:\[ 3{,}072 \rightarrow 4{,}704 \rightarrow 1{,}176 \rightarrow 1{,}600 \rightarrow 400 \rightarrow 120 \rightarrow 84 \rightarrow 10 \]

The count does not have to decrease at every individual layer. A convolution can increase the number of channels enough to increase the total number of activations.

However, the general trend across major network stages is toward a smaller and more compressed representation.

If the representation shrinks too quickly, useful spatial information may be lost before the network learns sufficiently rich features.

Spatial and Channel Trends

The spatial dimensions follow:\[ 32 \rightarrow 28 \rightarrow 14 \rightarrow 10 \rightarrow 5 \]

The channel dimensions follow:\[ 3 \rightarrow 6 \rightarrow 6 \rightarrow 16 \rightarrow 16 \]

Convolution increases the number of channels:\[ 3\rightarrow6\rightarrow16 \]

Pooling preserves the channel count while reducing spatial dimensions.

This pattern allows the model to replace fine spatial detail with a larger collection of increasingly abstract features.

Receptive-Field Growth

As the network becomes deeper, each activation depends on a larger region of the original image.

Let:

• \(r_l\) be the receptive-field size after layer \(l\).
• \(j_l\) be the spacing between adjacent receptive-field centers in the original image.

Initialize:\[ r_0=1, \qquad j_0=1 \]

For a layer with filter size \(f_l\) and stride \(s_l\):\[ r_l = r_{l-1} + (f_l-1)j_{l-1} \]\[ j_l = j_{l-1}s_l \]

Applying this to the network:

Each final \(5\times5\times16\) activation depends on an approximate \(16\times16\) region of the original image.

This illustrates how deeper features combine information from progressively larger image regions.

Layer-Counting Conventions

There are two common ways to count layers.

Count every operation

Under this convention:

• CONV 1 is one layer.
• POOL 1 is another layer.
• CONV 2 is another layer.
• POOL 2 is another layer.

Count only trainable layers

Under this convention, only operations containing learned parameters are counted:

• CONV 1
• CONV 2
• FC 3
• FC 4
• Output layer

Pooling and flattening are not counted because they contain no trainable parameters.

This difference explains why two sources may report different depths for the same architecture.

When comparing model depths, check whether pooling, activation, normalization, and output operations are included in the layer count.

Convolution–Pooling Blocks

Traditional CNNs often organize operations into blocks:\[ \text{CONV} \rightarrow \text{activation} \rightarrow \text{POOL} \]

A deeper model may use:\[ \text{CONV} \rightarrow \text{CONV} \rightarrow \text{POOL} \]

The pooling operation reduces the feature-map size after one or more convolutional transformations.

A general traditional architecture is:\[ \left[ \text{CONV} \rightarrow \text{POOL} \right] \times N \rightarrow \text{FC} \rightarrow \text{OUTPUT} \]

Modern architectures often use several convolutions before each downsampling step and may use strided convolutions instead of pooling.

Why Reuse Established Architectures?

CNN design involves many choices:

• Number of convolutional stages
• Filter sizes
• Channel counts
• Strides
• Padding
• Pooling placement
• Dense-layer widths
• Activation functions
• Normalization
• Regularization

These choices interact in complicated ways.

A practical strategy is to begin with an architecture that has already performed well on a related problem. It can then be adapted to:

• The input resolution
• The number of classes
• The available data
• Computational constraints
• The target domain

Established architectures provide tested design patterns and reduce the risk of choosing an ineffective configuration from scratch.

Training the Classifier

For a target class \(y\), the multiclass cross-entropy loss is:\[ \mathcal{L} = -\sum_{k=0}^{9} y_k\log\hat{y}_k \]

Because the target is one-hot encoded, this reduces to:\[ \mathcal{L} = -\log\hat{y}_{\text{true class}} \]

Backpropagation computes gradients for:

• CONV 1 filters and biases
• CONV 2 filters and biases
• FC 3 weights and biases
• FC 4 weights and biases
• Softmax-layer weights and biases

Pooling and flattening have no trainable parameters, but gradients still flow through them to earlier layers.

A Compact Implementation

A framework-style implementation might resemble:

import torch.nn as nn

model = nn.Sequential(
    nn.Conv2d(
        in_channels=3,
        out_channels=6,
        kernel_size=5,
        stride=1,
        padding=0,
    ),
    nn.ReLU(),

    nn.MaxPool2d(
        kernel_size=2,
        stride=2,
    ),

    nn.Conv2d(
        in_channels=6,
        out_channels=16,
        kernel_size=5,
        stride=1,
        padding=0,
    ),
    nn.ReLU(),

    nn.MaxPool2d(
        kernel_size=2,
        stride=2,
    ),

    nn.Flatten(),
    nn.Linear(5 * 5 * 16, 120),
    nn.ReLU(),

    nn.Linear(120, 84),
    nn.ReLU(),

    nn.Linear(84, 10),
)

import torch.nn as nn

model = nn.Sequential(
    nn.Conv2d(
        in_channels=3,
        out_channels=6,
        kernel_size=5,
        stride=1,
        padding=0,
    ),
    nn.ReLU(),

    nn.MaxPool2d(
        kernel_size=2,
        stride=2,
    ),

    nn.Conv2d(
        in_channels=6,
        out_channels=16,
        kernel_size=5,
        stride=1,
        padding=0,
    ),
    nn.ReLU(),

    nn.MaxPool2d(
        kernel_size=2,
        stride=2,
    ),

    nn.Flatten(),
    nn.Linear(5 * 5 * 16, 120),
    nn.ReLU(),

    nn.Linear(120, 84),
    nn.ReLU(),

    nn.Linear(84, 10),
)

The final layer outputs logits. A numerically stable cross-entropy implementation can combine softmax and the classification loss internally.

Architectural Limitations

Although this network is useful for understanding CNN structure, it reflects an early design style.

Potential limitations include:

• Large parameter concentration in dense layers
• No normalization layers
• No residual connections
• Aggressive dependence on valid convolutions
• Fixed input dimensions before flattening
• Relatively shallow feature extraction

Modern systems may replace parts of this architecture with:

• Same-padded convolutions
• Batch normalization
• Residual blocks
• Global average pooling
• Dropout
• Depthwise separable convolutions
• Attention mechanisms

The foundational ideas remain the same: local feature extraction, spatial reduction, channel expansion, and final classification.

Common Mistakes

Using 10 filters in CONV 2

The intended architecture uses 16 filters, producing:\[ 10\times10\times16 \]

Writing the spatial progression as 32 to 20

The correct progression is:\[ 32\rightarrow28\rightarrow14\rightarrow10\rightarrow5 \]

Forgetting that pooling preserves channels

POOL 1 produces:\[ 14\times14\times6 \]

not \(14\times14\).

POOL 2 produces:\[ 5\times5\times16 \]

Giving pooling trainable parameters

Traditional max pooling has no learned weights or biases.

Using the wrong CONV 2 filter depth

The second-layer filters have shape:\[ 5\times5\times6 \]

because the previous layer has six channels.

Miscounting the flattened features

The final pooling volume contains:\[ 5\cdot5\cdot16=400 \]

values.

Applying softmax before a combined cross-entropy function

Many implementations expect raw logits and calculate softmax internally for numerical stability.

Key Takeaway

A traditional convolutional classifier combines convolution, pooling, and fully connected layers:\[ 32\times32\times3 \rightarrow 28\times28\times6 \rightarrow 14\times14\times6 \rightarrow 10\times10\times16 \rightarrow 5\times5\times16 \rightarrow 400 \rightarrow 120 \rightarrow 84 \rightarrow 10 \]

Its main structural pattern is:

• Convolution learns local features and changes the channel count.
• Pooling reduces spatial dimensions without changing channels.
• Flattening converts the final feature volume into a vector.
• Fully connected layers combine the learned features.
• Softmax produces class probabilities.

The network contains 62,006 trainable parameters, with most of them located in the fully connected layers. As the model becomes deeper, spatial dimensions decrease, channel counts increase, and individual activations represent progressively larger regions of the original image.