Residual Networks and Skip Connections
Very deep neural networks can be difficult to optimize. As information and gradients pass through many layers, problems such as vanishing gradients, exploding gradients, and training degradation can make additional depth ineffective.
Residual Networks, commonly called ResNets, address this problem using shortcut connections. These connections allow activations to bypass several transformations and flow directly into deeper layers.
A ResNet does not remove the ordinary sequence of layers. It adds a second path through which information and gradients can travel.
This design made it practical to train networks containing more than 100 layers. The original ResNet research evaluated architectures as deep as 152 layers on ImageNet. Deep Residual Learning for Image Recognition
The Main Path Through Two Layers
Consider two consecutive layers of a neural network. The computation begins with the activation \(A^{[l]}\).
The first layer performs a linear transformation:\[ Z^{[l+1]} = W^{[l+1]}A^{[l]}+b^{[l+1]} \]
It then applies an activation function such as ReLU:\[ A^{[l+1]} = g\left(Z^{[l+1]}\right) \]
The next layer performs another linear transformation:\[ Z^{[l+2]} = W^{[l+2]}A^{[l+1]}+b^{[l+2]} \]
Finally, another activation function produces:\[ A^{[l+2]} = g\left(Z^{[l+2]}\right) \]
The complete computation is therefore:\[ A^{[l]} \rightarrow Z^{[l+1]} \rightarrow A^{[l+1]} \rightarrow Z^{[l+2]} \rightarrow A^{[l+2]} \]
This sequence is called the main path.
For information from \(A^{[l]}\) to reach \(A^{[l+2]}\), it must ordinarily pass through every linear transformation and nonlinear activation along this path.
Adding a Shortcut Connection
A residual block adds a direct connection from \(A^{[l]}\) to a deeper point in the network.
Instead of computing only:\[ A^{[l+2]} = g\left(Z^{[l+2]}\right) \]
the residual block computes:\[ A^{[l+2]} = g\left( Z^{[l+2]}+A^{[l]} \right) \]
The activation \(A^{[l]}\) is copied forward and added to \(Z^{[l+2]}\) before applying the final activation function.
Substituting the expression for \(Z^{[l+2]}\) gives:\[ A^{[l+2]} = g\left( W^{[l+2]}A^{[l+1]} +b^{[l+2]} +A^{[l]} \right) \]
The direct path from \(A^{[l]}\) to the addition operation is called:
- A shortcut connection
- A skip connection
- An identity connection, when it passes the input without transformation
These terms describe the same central idea: an activation can skip intermediate transformations and contribute directly to the output of a deeper layer.
Structure of a Residual Block
A residual block has two computational paths.
Main path
The main path applies the learned transformations:\[ F\left(A^{[l]}\right) = W^{[l+2]}g\left( W^{[l+1]}A^{[l]}+b^{[l+1]} \right)+b^{[l+2]} \]
In convolutional ResNets, these transformations are normally convolutional layers rather than ordinary matrix multiplications.
Shortcut path
The shortcut carries the original activation forward:\[ A^{[l]} \rightarrow A^{[l]} \]
Combined output
The two paths are added:\[ A^{[l+2]} = g\left( F\left(A^{[l]}\right)+A^{[l]} \right) \]
Using \(X\) for the block input, this is commonly written as:\[ Y=g\left(F(X)+X\right) \]
Here:
- \(X\) is the input carried by the shortcut.
- \(F(X)\) is the transformation learned by the main path.
- \(F(X)+X\) combines the two paths.
- \(g\) is the activation function following the addition.
What “Residual” Means
Suppose the desired mapping of a group of layers is \(H(X)\). A plain network attempts to learn that mapping directly:\[ X\rightarrow H(X) \]
A residual block instead represents the mapping as:\[ H(X)=F(X)+X \]
Therefore, the learned part is:\[ F(X)=H(X)-X \]
This difference between the desired output and the original input is called the residual.
The block does not need to construct the entire mapping from scratch. It can preserve the input through the shortcut and learn only the required modification.
Residual learning asks the main path to learn what should change, while the shortcut preserves what should remain unchanged.
Stacking Residual Blocks
A Residual Network is created by stacking many residual blocks.
A plain network might have the structure:\[ X \rightarrow \text{Layer 1} \rightarrow \text{Layer 2} \rightarrow \text{Layer 3} \rightarrow \text{Layer 4} \rightarrow \hat{Y} \]
A residual version adds shortcuts across groups of layers:\[ X \rightarrow \boxed{\text{Residual block 1}} \rightarrow \boxed{\text{Residual block 2}} \rightarrow \boxed{\text{Residual block 3}} \rightarrow \hat{Y} \]
Each residual block contains both a learned main path and a shortcut path.
A network with five residual blocks may contain approximately ten convolutional transformations inside those blocks, together with additional input and output layers.
The Degradation Problem in Plain Networks
In theory, a deeper network should be able to perform at least as well as a shallower network. The extra layers could simply learn identity mappings and preserve the behavior of the shallower model.
In practice, plain networks do not always behave this way.
As depth increases:
- Training error may initially decrease.
- Additional layers may improve representation capacity.
- Beyond a certain depth, optimization becomes more difficult.
- Training error may begin to increase.
This is called the degradation problem.
It is important to distinguish degradation from overfitting.
| Problem | Training error | Development or test error |
|---|---|---|
| Overfitting | Low | Much higher |
| Degradation | Higher than a shallower model | Often higher as a consequence |
| Underfitting | High | Usually high |
If a deeper network has greater training error than its shallower counterpart, the problem cannot be explained only by overfitting. The optimization procedure has failed to use the additional capacity effectively.
Why More Layers Can Be Difficult to Train
In a very deep plain network, both activations and gradients must repeatedly pass through many transformations.
During backpropagation, gradients contain products of many derivatives and weight matrices. Depending on their magnitudes, these products may become:
- Extremely small, causing vanishing gradients
- Extremely large, causing exploding gradients
- Poorly conditioned, making optimization unstable or slow
Modern initialization, normalization, and activation functions reduce these problems, but they do not fully solve the degradation problem in very deep plain networks.
Residual connections provide additional routes through which signals can propagate.
Shortcut Connections and Gradient Flow
Consider a simplified residual block without the final activation:\[ X_{l+1}=X_l+F(X_l,W_l) \]
Differentiating the output with respect to the input gives:\[ \frac{\partial X_{l+1}}{\partial X_l} = I+ \frac{\partial F(X_l,W_l)}{\partial X_l} \]
The derivative contains the identity term \(I\).
In a plain network, the gradient must depend entirely on derivatives through the learned transformations. In a residual block, an additional identity route is available.
This means that:
- Forward activations can pass directly to deeper layers.
- Backward gradients can propagate through shorter paths.
- The model does not depend exclusively on long chains of transformations.
- Earlier layers can receive stronger and more stable learning signals.
Residual connections do not make gradient problems impossible, but they substantially improve the conditions under which very deep networks are optimized.
Identity Shortcuts Require Matching Shapes
The addition\[ F(X)+X \]
is valid only if both tensors have the same dimensions.
For example, the following addition is valid:\[ F(X)\in\mathbb{R}^{28\times28\times64} \]\[ X\in\mathbb{R}^{28\times28\times64} \]
The output also has shape:\[ Y\in\mathbb{R}^{28\times28\times64} \]
For this reason, residual blocks frequently use stride-1 convolutions with same padding. These settings preserve height and width.
Projection Shortcuts
Sometimes the residual branch changes the spatial dimensions or number of channels.
For example:\[ X\in\mathbb{R}^{28\times28\times64} \]
while:\[ F(X)\in\mathbb{R}^{14\times14\times128} \]
These tensors cannot be added directly.
The shortcut must first transform \(X\):\[ Y=g\left(F(X)+W_sX\right) \]
The transformation \(W_s\) is commonly implemented using a \(1\times1\) convolution. A stride of 2 can reduce the spatial dimensions, while the number of filters changes the channel count.
In this example, the projection shortcut would use:
- A \(1\times1\) filter
- Stride 2
- 128 output channels
It transforms:\[ 28\times28\times64 \rightarrow 14\times14\times128 \]
The transformed shortcut and residual branch can then be added element by element.
Plain Networks and ResNets Compared
| Characteristic | Plain network | Residual network |
|---|---|---|
| Information path | Sequential only | Sequential path plus shortcuts |
| Learned mapping | Complete transformation | Residual correction |
| Identity mapping | Must be learned through several layers | Available through the shortcut |
| Gradient path | Must cross every intermediate transformation | Can follow shorter identity routes |
| Effect of extreme depth | Training may degrade | Much easier to optimize |
| Shortcut transformation | Not applicable | Identity or projection |
How Deep Can a ResNet Be?
The original ResNet study introduced models with 18, 34, 50, 101, and 152 layers. It also investigated networks with hundreds and even 1,000 layers on CIFAR-10.
Depth alone does not guarantee better practical performance. After some point:
- Accuracy improvements may become small.
- Training and inference become more expensive.
- Memory requirements increase.
- Wider or better-designed networks may be more efficient.
- Extremely deep models can still suffer from optimization or generalization limitations.
The significance of ResNet is therefore not that a network should always be made deeper. Its importance is that residual learning makes depths that were previously difficult to optimize much more manageable.
Original and Modern Residual Blocks
The block described above corresponds to the original post-activation formulation:\[ Y=\operatorname{ReLU}\left(F(X)+X\right) \]
Later ResNet variants moved normalization and activation operations before the convolutions. This produces a pre-activation block with a cleaner identity path:\[ Y=X+F(X) \]
Both designs use the same fundamental principle: the shortcut carries information around the learned residual transformation.
Key Takeaway
A Residual Network is built by adding shortcut connections across groups of neural-network layers. A residual block computes:\[ Y=g\left(F(X)+X\right) \]
rather than relying only on a long sequence of learned transformations.
The shortcut allows activations and gradients to travel through direct identity paths. This reduces the optimization difficulties associated with very deep plain networks and allows networks containing more than 100 layers to be trained successfully.
Most importantly, a residual block learns a correction \(F(X)\) to its input rather than learning the entire target transformation from scratch:\[ H(X)=X+F(X) \]
This simple architectural change is the foundation of ResNet.