Why Residual Networks Work So Well
Residual Networks, or ResNets, make it possible to train extremely deep neural networks without suffering as severely from the optimization problems found in ordinary deep networks.
The key idea is simple:
Instead of forcing a group of layers to learn an entirely new transformation, a residual block only needs to learn how its output should differ from its input.
This is accomplished through a shortcut connection that allows information to bypass one or more layers.
The original ResNet paper showed that residual networks were easier to optimize and successfully trained networks as deep as 152 layers on ImageNet. Deep Residual Learning for Image Recognition
The Degradation Problem in Deep Networks
Increasing the depth of a neural network should theoretically increase its representational capacity. A deeper network can represent everything a shallower network can, while potentially learning additional useful features.
In practice, however, simply adding more layers to a conventional network can make optimization harder. After a certain point, the deeper network may produce higher training error than a shallower version.
This is known as the degradation problem.
It is different from overfitting:
- Overfitting means training error is low while development or test error is high.
- Degradation means the deeper model cannot even achieve the training performance of the shallower model.
The problem is therefore not merely that the deeper model generalizes poorly. It may be difficult to optimize in the first place.
From a Plain Network to a Residual Network
Suppose a large neural network produces an activation \(A^{[l]}\). We then add two more layers:\[ A^{[l]} \rightarrow Z^{[l+1]} \rightarrow A^{[l+1]} \rightarrow Z^{[l+2]} \rightarrow A^{[l+2]} \]
In a plain network, the second new layer would compute:\[ A^{[l+2]} = g\left(Z^{[l+2]}\right) \]
where\[ Z^{[l+2]} = W^{[l+2]}A^{[l+1]}+b^{[l+2]} \]
A residual block introduces a shortcut connection from \(A^{[l]}\) directly to the input of the final activation:\[ A^{[l+2]} = g\left( Z^{[l+2]}+A^{[l]} \right) \]
Substituting the expression for \(Z^{[l+2]}\):\[ A^{[l+2]} = g\left( W^{[l+2]}A^{[l+1]} +b^{[l+2]} +A^{[l]} \right) \]
The shortcut carries the earlier activation forward without passing it through the two intermediate transformations.
Residual Formulation
A residual block is commonly written more compactly as:\[ Y = F(X;W)+X \]
Here:
- \(X\) is the block’s input.
- \(F(X;W)\) is the transformation learned by the residual branch.
- \(X\) is also passed through the shortcut connection.
- The two paths are added to produce \(Y\).
In the original post-activation ResNet design, an activation function is applied after the addition:\[ A_{\text{out}} = \operatorname{ReLU}\left(F(A_{\text{in}};W)+A_{\text{in}}\right) \]
Instead of learning a complete mapping \(H(X)\), the stacked layers learn a residual function:\[ F(X)=H(X)-X \]
The desired mapping can therefore be reconstructed as:\[ H(X)=F(X)+X \]
The Identity Function Is Easy to Represent
Consider what happens when the residual branch learns weights and biases close to zero:\[ W^{[l+2]} \approx 0 \]\[ b^{[l+2]} \approx 0 \]
The residual branch then contributes approximately zero:\[ F\left(A^{[l]}\right)\approx 0 \]
The block becomes:\[ A^{[l+2]} \approx g\left(A^{[l]}\right) \]
If \(g\) is ReLU and \(A^{[l]}\) is already nonnegative, then:\[ \operatorname{ReLU}\left(A^{[l]}\right) = A^{[l]} \]
Therefore:\[ A^{[l+2]} \approx A^{[l]} \]
The residual block has learned the identity function: it simply copies its input to its output.
If the additional layers do not learn anything useful, the shortcut gives the block a simple way to preserve the existing representation.
This explains why adding residual blocks is often much less damaging than adding ordinary layers to a plain network.
It is important, however, not to interpret this as an absolute guarantee. Optimization, initialization, normalization, and architecture choices can still affect performance. The identity-function argument explains why ResNets provide a much more favorable optimization structure.
Learning Improvements Over the Identity Mapping
The purpose of a residual block is not merely to copy its input. The identity mapping provides a convenient baseline from which the model can learn useful changes.
Suppose the desired transformation is:\[ H(X)=X+\Delta(X) \]
The residual branch only needs to learn:\[ F(X)=\Delta(X) \]
If no change is necessary, it can learn:\[ F(X)=0 \]
If a useful transformation is required, it learns only the correction that should be added to the input.
This is often easier than learning the complete transformation \(H(X)\) from scratch.
Why Shortcut Connections Improve Gradient Flow
Shortcut connections help not only with forward propagation but also with backward propagation.
Consider a simplified residual block:\[ X_{l+1}=X_l+F(X_l,W_l) \]
Differentiating with respect to \(X_l\) gives:\[ \frac{\partial X_{l+1}}{\partial X_l} = I+ \frac{\partial F(X_l,W_l)}{\partial X_l} \]
The gradient contains an identity term \(I\). This provides a direct route through which gradients can propagate backward.
For a sequence of residual blocks, the network can be written conceptually as:\[ X_L = X_l+ \sum_{i=l}^{L-1}F(X_i,W_i) \]
Thus, information from an earlier layer can reach a much later layer through the identity path, while gradients can travel backward through the same structure.
This does not completely eliminate vanishing or exploding gradients, but it makes optimization substantially easier. Later work on pre-activation ResNets emphasized that clean identity paths improve both forward and backward signal propagation. Identity Mappings in Deep Residual Networks
Matching Dimensions for Shortcut Addition
The two tensors in a residual addition must have the same shape.
If the residual branch produces:\[ F(X)\in \mathbb{R}^{H\times W\times C} \]
then the shortcut tensor must also have shape:\[ X\in \mathbb{R}^{H\times W\times C} \]
Only then is the following element-wise addition valid:\[ F(X)+X \]
Identity shortcut
When height, width, and channel count remain unchanged, the shortcut can directly copy the input:\[ Y=F(X)+X \]
This is called an identity shortcut.
Convolutional layers with stride 1 and same padding are commonly used inside these blocks because they preserve spatial dimensions:\[ H_{\text{out}}=H_{\text{in}} \]\[ W_{\text{out}}=W_{\text{in}} \]
Projection shortcut
If the input and output shapes differ, the shortcut must transform the input:\[ Y=F(X)+W_sX \]
Here, \(W_s\) represents a projection operation. In convolutional networks, it is usually implemented using a \(1\times1\) convolution.
For example, suppose the input has shape:\[ 28\times28\times128 \]
but the residual branch produces:\[ 14\times14\times256 \]
The shortcut can use a \(1\times1\) convolution with:
- 256 filters
- stride 2
This transforms the shortcut tensor into:\[ 14\times14\times256 \]
The addition is then well-defined:\[ \underbrace{F(X)}_{14\times14\times256} + \underbrace{W_sX}_{14\times14\times256} \]
Ignoring bias terms, the projection contains:\[ 1\times1\times128\times256 = 32{,}768 \]
learnable weights.
Zero-padding channels was also considered in the original ResNet work, but learned \(1\times1\) projections are a common solution when spatial resolution or channel count changes.
A Basic Residual Block
A common basic residual block contains two \(3\times3\) convolutions.
Its structure is approximately:\[ X \rightarrow \operatorname{Conv}_{3\times3} \rightarrow \operatorname{BN} \rightarrow \operatorname{ReLU} \rightarrow \operatorname{Conv}_{3\times3} \rightarrow \operatorname{BN} \]
The result is added to the shortcut:\[ Y=F(X)+X \]
The original ResNet design then applies ReLU:\[ A_{\text{out}}=\operatorname{ReLU}(Y) \]
In words:
- Transform the input through two convolutional layers.
- Preserve the original input through a shortcut.
- Add the two results.
- Apply the activation function.
Post-Activation and Pre-Activation Blocks
The original ResNet uses a post-activation structure:\[ X_{l+1} = \operatorname{ReLU}\left( X_l+F(X_l) \right) \]
A later refinement introduced the pre-activation residual block. Batch normalization and ReLU are applied before the convolutional operations, leaving the addition path cleaner:\[ X_{l+1} = X_l+F(X_l) \]
The pre-activation design makes the identity path more direct and can improve optimization in very deep networks.
| Design | Simplified formula | Main characteristic |
|---|---|---|
| Post-activation | \(X_{l+1}=\operatorname{ReLU}(X_l+F(X_l))\) | Activation follows the addition |
| Pre-activation | \(X_{l+1}=X_l+F(X_l)\) | Cleaner identity path between blocks |
Residual Networks for Images
A typical image-based ResNet begins with a convolutional stem and then passes the representation through multiple groups of residual blocks.
A simplified structure is:\[ \text{Image} \rightarrow \text{Initial convolution} \rightarrow \text{Residual blocks} \rightarrow \text{Downsampling} \rightarrow \text{More residual blocks} \rightarrow \text{Global average pooling} \rightarrow \text{Classifier} \]
Most basic residual blocks use \(3\times3\) convolutions. Within a stage:
- Spatial dimensions usually remain unchanged.
- The channel count remains unchanged.
- Identity shortcuts can therefore be used.
Between stages:
- Spatial resolution may be reduced.
- The number of channels may increase.
- A projection shortcut can match the new dimensions.
A typical progression might be:
| Stage | Spatial size | Channels | Shortcut |
|---|---|---|---|
| Input features | \(56\times56\) | 64 | — |
| Residual stage 1 | \(56\times56\) | 64 | Identity |
| Residual stage 2 | \(28\times28\) | 128 | Projection during transition |
| Residual stage 3 | \(14\times14\) | 256 | Projection during transition |
| Residual stage 4 | \(7\times7\) | 512 | Projection during transition |
Bottleneck Residual Blocks
Very deep ResNets commonly use a bottleneck block rather than two full \(3\times3\) convolutions.
A bottleneck block uses:\[ 1\times1 \rightarrow 3\times3 \rightarrow 1\times1 \]
The three operations serve different purposes:
- The first \(1\times1\) convolution reduces the number of channels.
- The \(3\times3\) convolution processes spatial information at the reduced width.
- The final \(1\times1\) convolution restores or expands the channel count.
This design reduces computation while allowing the network to contain many more layers.
For example:\[ 256 \rightarrow 64 \rightarrow 64 \rightarrow 256 \]
The expensive \(3\times3\) convolution operates on 64 channels instead of 256.
Plain Networks Versus Residual Networks
| Property | Plain network | Residual network |
|---|---|---|
| Layer objective | Learn a complete transformation | Learn a correction to the input |
| Identity mapping | May be difficult to learn | Naturally represented by a zero residual |
| Information flow | Must pass through every transformation | Can travel through shortcut paths |
| Gradient flow | More vulnerable to degradation with depth | Direct identity terms support propagation |
| Very deep training | Often difficult | Substantially easier |
| Dimension changes | Normal layer transformation | Requires a projection or adjusted shortcut |
What ResNets Do and Do Not Guarantee
Residual connections provide a strong architectural advantage, but they do not guarantee that every deeper network will outperform every shallower one.
Performance still depends on:
- The optimization algorithm
- Learning-rate scheduling
- Weight initialization
- Normalization
- Regularization
- Dataset size and quality
- Residual-block design
- Downsampling strategy
- Model width and depth
The accurate conclusion is:
Residual connections make very deep networks easier to optimize by providing identity paths and by letting blocks learn residual corrections.
This is stronger and more precise than claiming that additional residual layers can never hurt performance.
Key Takeaway
Residual Networks work well because shortcut connections make the identity mapping easy to represent. If additional layers have nothing useful to learn, their residual output can remain near zero, allowing the input to pass through almost unchanged.
At the same time, the residual branch can learn useful corrections:\[ H(X)=X+F(X) \]
These shortcut paths improve information and gradient propagation, reduce the degradation problem, and make networks with dozens or even hundreds of layers practical to train. When tensor dimensions change, a \(1\times1\) projection can transform the shortcut so that element-wise addition remains valid.