Three architectures played especially important roles in the development of convolutional neural networks:

1. LeNet-5 established the basic convolution–pooling–classifier pattern.
2. AlexNet demonstrated that large CNNs trained with GPUs and large datasets could transform visual recognition.
3. VGG-16 showed the power of architectural uniformity and depth using repeated \(3\times3\) convolutions.

Together, these networks illustrate the evolution from small digit-recognition systems to deep, large-scale image classifiers.

Architecture Overview

Architecture	Year	Typical input	Weight layers	Parameters	Key contribution
LeNet-5	1998	\(32\times32\times1\)	5–7 depending on counting	About 60,000	Established the classic CNN pattern
AlexNet	2012	Approximately \(224\)–\(227\) RGB	8	About 60 million	Large-scale CNN training with GPUs and ReLU
VGG-16	2014	\(224\times224\times3\)	16	About 138 million	Deep, uniform stacks of \(3\times3\) convolutions

The precise layer count depends on whether pooling, activation, and subsampling operations are counted. The names AlexNet and VGG-16 generally refer to trainable-layer conventions.

LeNet-5

LeNet-5 was designed by Yann LeCun and collaborators for handwritten and machine-printed character recognition. It became one of the earliest influential examples of a complete convolutional recognition system. Original LeNet-5 paper

LeNet-5 Input

LeNet-5 receives a grayscale image:\[ X\in\mathbb{R}^{32\times32\times1} \]

The original digit images were smaller than \(32\times32\), but they were placed inside a \(32\times32\) input field. This gave the network useful boundary space and made the dimensions work naturally through valid convolutions.

Simplified LeNet-5 Architecture

The central architecture can be summarized as:\[ 32\times32\times1 \]\[ \downarrow\quad \text{C1: }5\times5,\ 6\text{ filters} \]\[ 28\times28\times6 \]\[ \downarrow\quad \text{S2: }2\times2\text{ subsampling} \]\[ 14\times14\times6 \]\[ \downarrow\quad \text{C3: }5\times5,\ 16\text{ filters} \]\[ 10\times10\times16 \]\[ \downarrow\quad \text{S4: }2\times2\text{ subsampling} \]\[ 5\times5\times16 \]\[ \downarrow\quad \text{C5} \]\[ 120 \]\[ \downarrow\quad \text{F6} \]\[ 84 \]\[ \downarrow\quad \text{output} \]\[ 10 \]

C1: First Convolution

The first convolution uses:

• Six filters
• Filter size \(5\times5\)
• Stride 1
• No padding

The output size is:\[ \frac{32-5}{1}+1=28 \]

Therefore:\[ 32\times32\times1 \longrightarrow 28\times28\times6 \]

The six channels represent six learned feature maps.

S2: First Subsampling Layer

The next stage reduces the spatial dimensions by a factor of two:\[ 28\times28\times6 \longrightarrow 14\times14\times6 \]

A modern explanation often describes this as \(2\times2\) average pooling with stride 2.

The original implementation was slightly more complicated than modern parameter-free average pooling. Its subsampling maps included learned coefficients and biases before applying a nonlinear function.

C3: Second Convolution

The next convolution uses 16 output feature maps with \(5\times5\) spatial filters:\[ 14\times14\times6 \longrightarrow 10\times10\times16 \]

The spatial dimensions are:\[ 14-5+1=10 \]

An important historical detail is that the original C3 layer did not connect every output map to every S2 channel. It used a manually designed partial connection pattern.

This reduced computation and broke certain symmetries between feature maps. A modern simplified implementation would normally use standard convolution in which every filter spans all input channels.

S4: Second Subsampling Layer

Another \(2\times2\), stride-2 subsampling operation gives:\[ 10\times10\times16 \longrightarrow 5\times5\times16 \]

The volume contains:\[ 5\cdot5\cdot16=400 \]

activations.

C5 and F6

The original C5 stage used 120 filters covering the complete \(5\times5\times16\) input:\[ 5\times5\times16 \longrightarrow 1\times1\times120 \]

Because each filter covers the complete spatial extent, this behaves similarly to a fully connected transformation under the fixed input size.

The next layer contains 84 units:\[ 120\longrightarrow84 \]

The final classifier distinguishes 10 digit categories.

The Original Output Was Not Modern Softmax

A modern implementation would usually use:\[ 84\longrightarrow10 \]

followed by softmax:\[ \hat{y}_k = \frac{e^{z_k}} { \sum_{j=1}^{10}e^{z_j} } \]

The original LeNet-5 system used a different output design based on distances to class prototypes rather than the modern softmax classifier.

LeNet-5 Parameter Count

LeNet-5 contained approximately:\[ 60{,}000 \]

trainable parameters.

The exact count depends on whether the original partial connections and subsampling coefficients are reproduced or the architecture is implemented using modern simplified layers.

Historical Characteristics

Several details differ from current conventions:

• Average-style subsampling was used instead of max pooling.
• Squashing nonlinearities were used rather than ReLU.
• Some channel connections were manually restricted.
• Subsampling stages could contain learned scaling and bias values.
• The output classifier was not a standard softmax layer.

Despite these differences, the central architecture remains recognizable:\[ \text{CONV} \rightarrow \text{POOL} \rightarrow \text{CONV} \rightarrow \text{POOL} \rightarrow \text{CLASSIFIER} \]

Lasting Design Patterns from LeNet-5

LeNet-5 established several patterns that remained influential:

• Spatial dimensions decrease with depth.
• Channel counts increase with depth.
• Convolutions learn local features.
• Pooling reduces spatial resolution.
• High-level features feed a classifier.

Its spatial progression is:\[ 32 \rightarrow 28 \rightarrow 14 \rightarrow 10 \rightarrow 5 \]

Its channel progression is:\[ 1 \rightarrow 6 \rightarrow 16 \]

AlexNet

AlexNet was developed by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. It demonstrated the effectiveness of training a large CNN on the ImageNet dataset using GPUs. The network contained five convolutional layers, three fully connected layers, and approximately 60 million parameters. Original AlexNet paper

Why AlexNet Was Important

AlexNet combined several ideas at an effective scale:

• A much larger training dataset
• A much larger CNN
• GPU-based training
• ReLU activations
• Data augmentation
• Dropout in fully connected layers
• Overlapping max pooling

Its performance helped establish deep convolutional networks as a dominant approach to large-scale visual recognition.

Input-Size Detail

The original paper describes inputs of:\[ 224\times224\times3 \]

However, a commonly presented dimension calculation uses:\[ 227\times227\times3 \]

because an \(11\times11\) filter with stride 4 and no padding produces:\[ \left\lfloor \frac{227-11}{4} \right\rfloor+1 = 55 \]

A \(224\times224\) input can also produce \(55\times55\) if appropriate padding is included:\[ \left\lfloor \frac{224+2(2)-11}{4} \right\rfloor+1 = 55 \]

This explains why descriptions and implementations sometimes use different input dimensions while producing the same first feature-map size.

Simplified AlexNet Architecture

A commonly used representation is:\[ 227\times227\times3 \]\[ \downarrow\quad \text{CONV 1: }11\times11,\ 96,\ s=4 \]\[ 55\times55\times96 \]\[ \downarrow\quad \text{MAX POOL} \]\[ 27\times27\times96 \]\[ \downarrow\quad \text{CONV 2: }5\times5,\ 256 \]\[ 27\times27\times256 \]\[ \downarrow\quad \text{MAX POOL} \]\[ 13\times13\times256 \]\[ \downarrow\quad \text{CONV 3: }3\times3,\ 384 \]\[ 13\times13\times384 \]\[ \downarrow\quad \text{CONV 4: }3\times3,\ 384 \]\[ 13\times13\times384 \]\[ \downarrow\quad \text{CONV 5: }3\times3,\ 256 \]\[ 13\times13\times256 \]\[ \downarrow\quad \text{MAX POOL} \]\[ 6\times6\times256 \]\[ \downarrow\quad \text{FLATTEN} \]\[ 9{,}216 \]\[ \downarrow\quad \text{FC} \]\[ 4{,}096 \]\[ \downarrow\quad \text{FC} \]\[ 4{,}096 \]\[ \downarrow\quad \text{SOFTMAX} \]\[ 1{,}000 \]

Correcting the Second-Layer Channel Count

The second convolution produces 256 channels:\[ 27\times27\times256 \]

It does not produce 276 channels.

CONV 1

The first layer uses:

• 96 filters
• \(11\times11\) spatial size
• Stride 4

The large filter and stride rapidly reduce the spatial resolution:\[ 227\times227\times3 \longrightarrow 55\times55\times96 \]

Compared with later architectural styles, this is an aggressive initial transformation.

Overlapping Max Pooling

AlexNet used \(3\times3\) max pooling with stride 2.

Because:\[ f=3>s=2 \]

adjacent pooling windows overlap.

The first pooling transformation is:\[ 55\times55\times96 \longrightarrow 27\times27\times96 \]

The second is:\[ 27\times27\times256 \longrightarrow 13\times13\times256 \]

The final pooling operation gives:\[ 13\times13\times256 \longrightarrow 6\times6\times256 \]

ReLU Activations

AlexNet used the rectified linear unit:\[ \operatorname{ReLU}(z)=\max(0,z) \]

ReLU avoids the saturation behavior associated with sigmoid and tanh for positive inputs and helped make training substantially faster.

The AlexNet paper explicitly highlighted non-saturating neurons as an important part of its system.

Fully Connected Classifier

The last convolutional representation contains:\[ 6\cdot6\cdot256 = 9{,}216 \]

values.

These are flattened and passed through two large fully connected hidden layers:\[ 9{,}216 \rightarrow 4{,}096 \rightarrow 4{,}096 \rightarrow 1{,}000 \]

The final layer classifies the image into one of 1,000 ImageNet categories.

A large proportion of AlexNet’s approximately 60 million parameters is concentrated in these fully connected layers.

Dropout

AlexNet applied dropout to its fully connected layers.

During training, dropout randomly suppresses some activations, reducing co-adaptation and helping control overfitting.

This was particularly important because the dense classification head contained tens of millions of parameters.

Data Augmentation

AlexNet also used image transformations to enlarge the effective training distribution, including:

• Translations and image crops
• Horizontal reflections
• Color-intensity modifications

These transformations improved generalization without requiring additional manually labeled images.

Two-GPU Design

The original network was divided across two GPUs because of the hardware limitations available at the time.

Some layers communicated across the two GPU partitions, while other connections remained within one partition. This led to grouped connectivity patterns in parts of the model.

The design is historically important but is not essential to the conceptual AlexNet architecture. Modern implementations commonly represent the model on one device or distribute it using more general parallel-computing systems.

Local Response Normalization

AlexNet included local response normalization, or LRN.

Conceptually, LRN normalized activations across nearby channels at a fixed spatial position. It was motivated by a form of lateral competition among feature maps.

LRN is not a standard component of most modern CNN architectures. Later work found other methods—especially batch normalization and improved architectural designs—to be more useful.

AlexNet Parameter Count

AlexNet contained approximately:\[ 60\text{ million parameters} \]

and roughly:\[ 650{,}000\text{ neurons} \]

according to its original description.

This was approximately a thousand times more parameters than LeNet-5.

AlexNet’s Historical Impact

AlexNet showed that familiar CNN principles could scale when combined with:

• Large labeled datasets
• Greater model capacity
• GPU acceleration
• ReLU
• Strong regularization
• Data augmentation

Its ImageNet results gave the broader vision community compelling evidence that deep convolutional networks could substantially outperform established approaches.

VGG-16

VGG-16 was developed by Karen Simonyan and Andrew Zisserman at the University of Oxford’s Visual Geometry Group.

Its main contribution was a simple and systematic architecture built primarily from:

• \(3\times3\) convolutions
• Stride 1
• Same padding
• \(2\times2\) max pooling with stride 2

The VGG study investigated networks with 16–19 trainable layers and showed that increasing depth with small filters could substantially improve recognition performance. Original VGG paper

VGG-16 Architecture

The architecture begins with:\[ 224\times224\times3 \]

It uses five convolutional blocks.

Block 1

Two convolutional layers with 64 filters:\[ 224\times224\times3 \]\[ \downarrow\quad 3\times3,\ 64 \]\[ 224\times224\times64 \]\[ \downarrow\quad 3\times3,\ 64 \]\[ 224\times224\times64 \]

Then max pooling:\[ 224\times224\times64 \longrightarrow 112\times112\times64 \]

Block 2

Two convolutional layers with 128 filters:\[ 112\times112\times64 \]\[ \downarrow\quad 3\times3,\ 128 \]\[ 112\times112\times128 \]\[ \downarrow\quad 3\times3,\ 128 \]\[ 112\times112\times128 \]

Then max pooling:\[ 112\times112\times128 \longrightarrow 56\times56\times128 \]

Block 3

Three convolutional layers with 256 filters:\[ 56\times56\times128 \]\[ \downarrow\quad 3\times3,\ 256 \]\[ 56\times56\times256 \]

Two more \(3\times3\) convolutions preserve the shape:\[ 56\times56\times256 \]

Then max pooling:\[ 56\times56\times256 \longrightarrow 28\times28\times256 \]

Block 4

Three convolutional layers with 512 filters:\[ 28\times28\times256 \longrightarrow 28\times28\times512 \]

The next two convolutions preserve:\[ 28\times28\times512 \]

Then max pooling:\[ 28\times28\times512 \longrightarrow 14\times14\times512 \]

Block 5

Three more convolutional layers with 512 filters:\[ 14\times14\times512 \]

The final max-pooling layer produces:\[ 7\times7\times512 \]

Classification Head

The final feature volume contains:\[ 7\cdot7\cdot512 = 25{,}088 \]

values.

The original VGG-16 classifier uses:\[ 25{,}088 \rightarrow 4{,}096 \rightarrow 4{,}096 \rightarrow 1{,}000 \]

The final 1,000 outputs correspond to ImageNet classes.

VGG-16 Summary Table

Block	Convolutional layers	Channels	Output after pooling
Block 1	2	64	\(112\times112\times64\)
Block 2	2	128	\(56\times56\times128\)
Block 3	3	256	\(28\times28\times256\)
Block 4	3	512	\(14\times14\times512\)
Block 5	3	512	\(7\times7\times512\)

The dense head is:\[ 25{,}088 \rightarrow 4{,}096 \rightarrow 4{,}096 \rightarrow 1{,}000 \]

Why It Is Called VGG-16

VGG-16 contains 16 trainable layers:

• 13 convolutional layers
• 3 fully connected layers

Thus:\[ 13+3=16 \]

Pooling and activation operations are not included in this count.

VGG-19 extends the same design with additional convolutional layers.

The Uniform Design Principle

VGG’s architecture is much more regular than AlexNet’s.

Nearly every convolution uses:\[ 3\times3,\qquad s=1,\qquad p=1 \]

Every pooling layer uses:\[ 2\times2,\qquad s=2 \]

This creates a predictable pattern:

• Convolutions preserve spatial dimensions.
• Pooling halves height and width.
• Channel counts increase after major reductions in resolution.

Spatial Progression

The height and width follow:\[ 224 \rightarrow 112 \rightarrow 56 \rightarrow 28 \rightarrow 14 \rightarrow 7 \]

Every pooling layer reduces the spatial dimensions by a factor of two.

Channel Progression

The channel count follows:\[ 3 \rightarrow 64 \rightarrow 128 \rightarrow 256 \rightarrow 512 \rightarrow 512 \]

The number of channels approximately doubles as spatial resolution decreases, until reaching 512.

This balances computational cost: smaller spatial maps can support more feature channels.

Why Stack Small \(3\times3\) Filters?

Two consecutive \(3\times3\) convolutions have an effective receptive field of:\[ 5\times5 \]

Three consecutive \(3\times3\) convolutions have an effective receptive field of:\[ 7\times7 \]

This provides an alternative to using one large filter.

Two \(3\times3\) layers vs. one \(5\times5\) layer

Assume the input and output both have \(C\) channels.

One \(5\times5\) convolution uses approximately:\[ 25C^2 \]

weights.

Two \(3\times3\) convolutions use:\[ 2(9C^2)=18C^2 \]

weights.

Thus, stacked \(3\times3\) layers use fewer parameters:\[ 18C^2<25C^2 \]

They also insert two nonlinear activation functions instead of one, increasing the network’s representational power.

Three \(3\times3\) layers vs. one \(7\times7\) layer

One \(7\times7\) convolution uses:\[ 49C^2 \]

weights.

Three \(3\times3\) convolutions use:\[ 3(9C^2)=27C^2 \]

weights.

Again:\[ 27C^2<49C^2 \]

The stacked design uses fewer weights and more nonlinear transformations.

VGG-16 Parameter Count

VGG-16 contains approximately:\[ 138\text{ million parameters} \]

Although its convolutional structure is elegant, the original dense classification head is extremely large.

The first fully connected layer alone contains approximately:\[ 25{,}088\cdot4{,}096 \approx 102.8\text{ million} \]

weights.

Thus, most VGG-16 parameters are located in its fully connected layers rather than its convolutional blocks.

Strengths of VGG-16

• Simple and uniform architecture
• Easy-to-understand spatial progression
• Strong learned visual representations
• Reusable convolutional features
• Influential template for later architectures

Limitations of VGG-16

• Very large parameter count
• High memory consumption
• Expensive computation
• Large dense classifier
• No residual connections
• Difficult to train compared with later architectures of similar depth

VGG remains historically and conceptually important even though more efficient networks are generally preferred for new systems.

Comparing LeNet-5, AlexNet, and VGG-16

Structural Comparison

Property	LeNet-5	AlexNet	VGG-16
Primary task	Digit recognition	Large-scale image classification	Large-scale image classification
Input channels	1	3	3
Main filter sizes	\(5\times5\)	\(11\times11\), \(5\times5\), \(3\times3\)	Almost entirely \(3\times3\)
Pooling	Average-style subsampling	Overlapping max pooling	\(2\times2\) max pooling
Activation	Squashing nonlinearities	ReLU	ReLU
Conv layers	3 under the original naming	5	13
Dense layers	Small classifier	Large classifier	Very large classifier
Parameters	About 60K	About 60M	About 138M

Evolution of Scale

The parameter counts increased dramatically:\[ 60{,}000 \rightarrow 60{,}000{,}000 \rightarrow 138{,}000{,}000 \]

This reflects improvements in:

• Available training data
• GPU computation
• Optimization
• Regularization
• Software
• Architectural understanding

Evolution of Activation Functions

LeNet-5 used saturating nonlinearities.

AlexNet popularized ReLU for large-scale vision:\[ g(z)=\max(0,z) \]

VGG-16 continued using ReLU throughout its deep convolutional stacks.

Evolution of Architectural Regularity

LeNet established the basic alternating pattern.

AlexNet scaled the design but used several different filter sizes and hardware-specific connection groups.

VGG simplified the design around repeated:\[ 3\times3\text{ convolution} \]

and:\[ 2\times2\text{ max pooling} \]

This regularity made VGG particularly easy to understand and adapt.

Common Patterns Across All Three Networks

Despite their differences, these networks share several principles.

Spatial dimensions decrease

\[ n_H,\ n_W \quad\text{decrease with depth} \]

This reduces computation and allows later features to summarize larger portions of the image.

Channel counts increase

\[ n_C \quad\text{generally increases with depth} \]

This allows later layers to represent a richer collection of abstract features.

Convolutional blocks precede classification

Early and middle layers extract spatial features. Later layers combine those features to make predictions.

Features become increasingly abstract

A simplified hierarchy is:\[ \text{edges} \rightarrow \text{textures} \rightarrow \text{parts} \rightarrow \text{objects} \]

Pooling separates major stages

Pooling or another downsampling operation marks transitions between spatial scales.

Practical Lessons from Classic Architectures

Reuse successful patterns

Designing every architectural detail from scratch is rarely necessary. Established designs provide reliable starting points.

Preserve spatial resolution early

Aggressive early downsampling can discard useful detail, particularly for small objects or high-resolution prediction tasks.

Increase channels as resolution decreases

This pattern allows the network to represent more feature types while keeping computational cost manageable.

Prefer simple repeated blocks

VGG demonstrated that a uniform design can be easier to analyze, implement, and modify than a network with many unrelated filter configurations.

Watch the dense classifier

Flattening a large feature volume into a fully connected layer can create most of the model’s parameters.

Modern systems often use global average pooling to avoid this:\[ n_H\times n_W\times n_C \longrightarrow 1\times1\times n_C \]

Historical details are not necessarily current defaults

Components such as:

• Local response normalization
• Partial channel connectivity
• Learned subsampling coefficients
• Large dense classifiers

are historically important but are not required in modern CNNs.

Key Takeaway

LeNet-5, AlexNet, and VGG-16 represent three major stages in CNN development.

LeNet-5 established the foundational pattern:\[ \text{CONV} \rightarrow \text{POOL} \rightarrow \text{CONV} \rightarrow \text{POOL} \rightarrow \text{CLASSIFIER} \]

AlexNet showed that CNNs could scale to large datasets using GPUs, ReLU, dropout, augmentation, and a much larger model.

VGG-16 introduced a highly regular architecture based on repeated \(3\times3\) convolutions and \(2\times2\) pooling:\[ 224 \rightarrow 112 \rightarrow 56 \rightarrow 28 \rightarrow 14 \rightarrow 7 \]

while channels increase:\[ 64 \rightarrow 128 \rightarrow 256 \rightarrow 512 \]

The broad design pattern remains influential: decrease spatial dimensions gradually, increase feature channels, and build complex visual representations by stacking simple convolutional operations.