CNN Architectures Compared: LeNet to EfficientNet

CNN families explore several related problems: learning useful features, training deeper networks, reducing arithmetic, and spending a larger compute budget. Their histories overlap; Inception and VGG, for example, explored different designs around the same time. Comparing the operations and constraints explains why several families remain useful without treating one as the inevitable replacement for another.

LeNet-5 (1998): the template

LeNet-5 has roughly 60,000 trainable parameters and seven layers after the input under the original paper’s counting convention. It combines convolutions, subsampling, and later classification layers for handwritten digits. The original subsampling includes learned scaling and bias, and its output uses distance-based units; it is not identical to a modern reconstruction with fixed average pooling and a softmax head. It demonstrated an end-to-end learned visual system at a much smaller image and task scale than ImageNet.

Convolution, padding, stride, and pooling are introduced in CNN Fundamentals: From Convolution to Image Classification.

AlexNet (2012): scale and ReLU

AlexNet combined five convolutional and three fully connected layers, about 60 million parameters, and training on two GPUs. ReLU helped optimization relative to saturating activations, while dropout, augmentation, and the large labeled dataset also contributed. ReLU still has a zero-gradient region for negative inputs and does not eliminate all optimization problems. The paper’s ILSVRC-2012 competition entry reported 15.3% top-5 error versus 26.2% for the runner-up; that ensemble result is not a single-checkpoint top-1 accuracy.

The three dense layers account for about 58.6 million parameters in the 6×6×256-to-4,096-to-4,096-to-1,000 head. That makes the head a major memory cost. It also performs learned nonlinear classification; a large parameter count does not show that it does little representational work. Replacing it with pooling changes the model’s capacity and its use of spatial layout.

VGG (2014): uniformity

The widely used VGG-16 and VGG-19 configurations stack 3×3 stride-1 convolutions with padding 1 and 2×2 stride-2 max pooling. Stage widths are 64, 128, 256, 512, and 512, so they do not double after every pool. The paper also tested a configuration with 1×1 layers. With constant channel width and biases excluded, two 3×3 layers use \(18C^2\) weights versus \(25C^2\) for one 5×5 layer, while reaching the same nominal receptive field and allowing an intervening nonlinearity.

VGG-16 with its 1,000-class dense head has about 138 million parameters. Its regular structure makes stages easy to inspect, but the dense head and high-resolution convolutions can be expensive. Feature-extractor use omits or bypasses the classification head, changing that cost. The ResNet paper’s degradation experiments should not be read as a universal VGG depth limit of 19 layers.

ResNet (2015): the identity shortcut

The ResNet paper compared plain networks and observed higher training error in a 56-layer network than a 20-layer one on CIFAR-10. Higher training error points to a fitting or optimization difficulty, rather than establishing overfitting from a train–test gap. The identity-extension argument motivates the design: if added layers can preserve the required representation, extra depth need not worsen the best attainable fit. Architectural details still determine which exact identity mappings are representable.

A shape-preserving residual block learns a correction F(x) to its input x. Before any post-addition activation, it has the form

\[y=F(x,\{W_i\})+x\]

For this map, \(J_y=J_F+I\), where \(J\) denotes the Jacobian and \(I\) the identity matrix. Backpropagation sends an incoming gradient \(g\) to \(g+J_F^\top g\). If \(J_F\) is small, one block is locally close to the identity, but the contributions can also cancel: \(F(x)=-x\) gives zero derivative. Products of many near-identity Jacobians can still shrink or grow. A shortcut improves the optimization structure; it does not guarantee undiminished gradients or successful training at every depth.

import torch
import torch.nn as nn

class ResidualBlock(nn.Module):
    def __init__(self, c_in, c_out, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(c_in, c_out, 3, stride, 1, bias=False)
        self.bn1 = nn.BatchNorm2d(c_out)
        self.conv2 = nn.Conv2d(c_out, c_out, 3, 1, 1, bias=False)
        self.bn2 = nn.BatchNorm2d(c_out)
        self.relu = nn.ReLU(inplace=True)
        # Projection only when shape changes; identity otherwise.
        self.shortcut = nn.Sequential()
        if stride != 1 or c_in != c_out:
            self.shortcut = nn.Sequential(
                nn.Conv2d(c_in, c_out, 1, stride, bias=False),
                nn.BatchNorm2d(c_out))

    def forward(self, x):
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        return self.relu(out + self.shortcut(x))    # add BEFORE the activation

print(ResidualBlock(64, 128, stride=2)(torch.randn(2, 64, 32, 32)).shape)
# torch.Size([2, 128, 16, 16])

This code is a post-activation basic block: y = ReLU(F(x) + S(x)). When shape is unchanged, S is the identity; when channels or stride change, S is a learned projection. The post-addition ReLU can gate gradients, and a projection has its own Jacobian instead of I. Pre-activation variants place normalization and activations before the residual-branch convolutions and omit the post-addition activation for a cleaner identity shortcut. Moving activations inside the branch does not itself break the shortcut. Also, F is not generally zero at random initialization, so the block need not start as the identity.

Inception (2014–2016): width and factorization

An early Inception module sends the same feature map through several branches: a 1×1 convolution, reduced-width 3×3 and 5×5 paths, and a pooling path. Their spatial sizes are aligned and outputs are concatenated along the channel axis. If the branches output 64, 128, 32, and 32 channels, concatenation produces 256 channels, preserving each branch’s outputs for later mixing. This differs from residual addition, which combines matching entries and keeps the same channel count.

Inception’s 1×1 reductions precede the expensive 3×3 and 5×5 operations; the pooling path commonly projects after pooling, and the direct 1×1 branch is already a pointwise path. Later variants use stacks of smaller or asymmetric filters. These factorizations constrain or change the computation and can add nonlinearities; they are not exact replacements for every arbitrary large kernel. GoogLeNet avoids VGG’s large dense head and uses far fewer parameters, but totals depend on auxiliary classifiers and implementation. Branch alignment and operator scheduling add engineering considerations when modifying or deploying the network.

MobileNet (2017): depthwise separable convolution

MobileNetV1 separates spatial filtering from channel mixing. With depth multiplier 1, the depthwise operation applies one f×f filter to each input channel; a pointwise 1×1 layer mixes the resulting channels. For matching output spatial size, excluding biases, normalization, and activations, the separable-to-standard ratio of multiply-accumulates (MACs) is

\[\frac{f^2C_{in}+C_{in}C_{out}}{f^2C_{in}C_{out}}=\frac{1}{C_{out}}+\frac{1}{f^2}\]

For a 3×3 filter and 256 output channels, the standard operation uses about 8.69 times as many MACs. The ratio depends on output width; it is not eightfold for every layer. This is a more constrained mapping than a general standard convolution, so equal input and output shapes do not mean equal expressive power. A MAC counts one multiplication with accumulation; a convention counting multiplication and addition separately reports about twice as many FLOPs.

def standard_cost(f, c_in, c_out, hw):
    return f * f * c_in * c_out * hw * hw

def separable_cost(f, c_in, c_out, hw):
    return (f * f * c_in * hw * hw) + (c_in * c_out * hw * hw)

s = standard_cost(3, 128, 256, 28)
d = separable_cost(3, 128, 256, 28)
print(s, d, round(s / d, 2))          # 231211008 26593280 8.69

MobileNetV2 expands channels with a pointwise operation, filters them depthwise, then projects back to a narrow representation. Its linear bottleneck means there is no ReLU after that projection, helping avoid discarding negative components in the narrow representation. A residual connection joins the narrow endpoints only when their shapes match, normally at stride 1. This inverted residual differs from the wide-end shortcuts of a ResNet bottleneck. The V1 MAC formula above does not include V2’s extra expansion stage.

EfficientNet (2019): compound scaling

EfficientNet scales a searched baseline along depth, channel width, and input resolution using multipliers \(d=\alpha^\phi\), \(w=\beta^\phi\), and \(r=\gamma^\phi\). Here \(\phi\) controls the scale, and \(\alpha,\beta,\gamma\) specify how it is divided among the three dimensions. For the approximate convolutional cost relation \(d w^2 r^2\), choosing \(\alpha\beta^2\gamma^2\approx2\) gives a cost multiplier near \(2^\phi\). Increasing \(\phi\) by one approximately doubles cost; doubling \(\phi\) generally does not. For example, φ = 1, 2, and 4 correspond approximately to 2×, 4×, and 16× cost. Channel and layer rounding and different operators make this an approximation.

The original B0 baseline uses mobile inverted bottleneck blocks with squeeze-and-excitation, which pools channel information and learns channel-wise gates. Compound scaling builds a family around that baseline; it is not a theorem that every architecture should grow in those proportions. The paper reports favorable accuracy–cost comparisons under its experiments. Deployment latency also depends on memory traffic, operator support, precision, fusion, and batch size; a lower MAC count need not give a faster model.

Comparison

An accuracy comparison needs named checkpoints and a common protocol: dataset split, input resolution, crop policy, and single-model versus ensemble evaluation. Top-1 accuracy asks whether the highest-scoring class is correct; top-5 allows any of the five highest-scoring classes. Record training or fine-tuning budgets and preprocessing too. Parameter count measures stored trainable values, MACs estimate arithmetic, and measured latency describes a particular implementation and device. None substitutes for the others.

Alternatives that use attention for image modeling are covered in Vision Transformers and Modern Vision Models. CNNs and attention-based models can also be combined.

Choosing one

• Establish a relevant baseline. A supported pretrained ResNet is one practical candidate, not a mandatory winner. Check source weights, target preprocessing, and whether its feature stages fit the task.
• Measure deployment constraints. Compare mobile-oriented and standard convolutions at the actual batch size, precision, and input resolution. Include peak memory and representative latency, not only MACs.
• Compare adaptation plans. Small target datasets do not prove that the smallest pretrained model wins. Compare suitable checkpoints and freezing or fine-tuning choices under a stated budget.
• Match outputs to the task. Detection and segmentation may need several feature resolutions. Check their channel widths and strides and the downstream head, rather than choosing solely by image-classification accuracy.

Residual addition and position-wise channel transformations recur in later models. Transformer feed-forward layers often expand and contract the feature dimension independently at each token, which is related to the shared channel mixing of a 1×1 convolution. That analogy does not mean every transformer literally contains a convolutional bottleneck. Choose a model by testing the whole training and deployment setup, including the required prediction quality.

A practical fine-tuning and augmentation recipe is in Image Classification in Practice: Transfer Learning and Augmentation.

Exercises

1. Where the dense-head parameters live. Count an AlexNet-style head mapping 6×6×256 features to 4,096, then 4,096, then 1,000 outputs, with biases. Compare the whole head with global average pooling followed by a 256-to-1,000 classifier. What does this saving not establish?

fc1 = 6*6*256*4096 + 4096
fc2 = 4096*4096 + 4096
fc3 = 4096*1000 + 1000
original_head = fc1 + fc2 + fc3
gap_head = 256*1000 + 1000
print(fc1, original_head, gap_head, original_head - gap_head)
# 37752832 58631144 257000 58374144

2. Depthwise separable, counted. Compute MACs for a standard 3×3 convolution from 128 to 256 channels with a 28×28 output and a depthwise–pointwise alternative with the same spatial sizes. Use depth multiplier 1 and ignore biases and other operations. Compare the ratio with the formula and state its limits.

def standard(f, ci, co, hw):
    return f*f*ci*co*hw*hw

def separable(f, ci, co, hw):
    return (f*f*ci + ci*co)*hw*hw

s, d = standard(3,128,256,28), separable(3,128,256,28)
print(s, d, round(s/d, 2))
print(round(1/(1/256 + 1/9), 2))
# 231211008 26593280 8.69
# 8.69

3. Read the residual gradient. Differentiate y = x + F(x). Compare F(x) = 0 and F(x) = −x, then test a post-addition ReLU at x = −1 with F(x) = 0. Does an identity summand guarantee a nonzero gradient through the whole block?

import torch

for name, fn in [
    ("identity", lambda x: x + 0*x),
    ("cancellation", lambda x: x - x),
    ("post_relu", lambda x: torch.relu(x + 0*x)),
]:
    x = torch.tensor(-1.0, requires_grad=True)
    fn(x).backward()
    print(name, x.grad.item())
# identity 1.0
# cancellation 0.0
# post_relu 0.0

References

• He et al. (2016). Identity Mappings in Deep Residual Networks.
• LeCun, Bottou, Bengio, and Haffner (1998). Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE.
• Krizhevsky, Sutskever, and Hinton (2012). ImageNet Classification with Deep Convolutional Neural Networks. NIPS.
• Simonyan and Zisserman (2015). Very Deep Convolutional Networks for Large-Scale Image Recognition.
• Szegedy et al. (2015). Going Deeper with Convolutions. CVPR.
• He, Zhang, Ren, and Sun (2016). Deep Residual Learning for Image Recognition. CVPR.
• Howard et al. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv preprint.
• Sandler, Howard, Zhu, Zhmoginov, and Chen (2018). MobileNetV2: Inverted Residuals and Linear Bottlenecks.
• Tan and Le (2019). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. ICML.