Transposed Convolution for Learnable Upsampling
Transposed convolution is a learnable operation that increases the spatial resolution of a feature map. It is commonly used in semantic-segmentation decoders, including U-Net, where a compressed representation must be expanded back toward the input image’s resolution.
For example, a transposed convolution can transform:\[ 2\times2 \longrightarrow 4\times4 \]
Unlike ordinary interpolation, the transformation is controlled by trainable kernel weights.
Ordinary and Transposed Convolution
An ordinary convolution typically reduces or preserves spatial resolution.
For example:\[ 6\times6\times3 \]
convolved with five filters of shape:\[ 3\times3\times3 \]
using stride one and no padding produces:\[ 4\times4\times5 \]
A transposed convolution reverses the direction of this spatial transformation:\[ 2\times2 \longrightarrow 4\times4 \]
It distributes each input value over a larger output region using a learned filter.
Transposed convolution expands spatial dimensions, but it is not generally an inverse convolution.
Information removed by an earlier convolution or pooling layer cannot usually be reconstructed exactly.
The Core Operation
Suppose the input is:\[ X= \begin{bmatrix} x_{11} & x_{12}\\ x_{21} & x_{22} \end{bmatrix} \]
and the learned kernel is:\[ K= \begin{bmatrix} k_{11} & k_{12} & k_{13}\\ k_{21} & k_{22} & k_{23}\\ k_{31} & k_{32} & k_{33} \end{bmatrix} \]
For every input value:
- Multiply the complete kernel by that input value.
- Place the resulting weighted kernel into the output canvas.
- Move by the specified stride.
- Add values wherever placed kernels overlap.
Each input element therefore contributes to several output positions.
A Simple Example Without Cropping
Consider:\[ X= \begin{bmatrix} 2 & 1\\ 3 & 2 \end{bmatrix} \]
with the kernel:\[ K= \begin{bmatrix} 0 & 1 & 0\\ 1 & 2 & 1\\ 0 & 1 & 0 \end{bmatrix} \]
Use stride:\[ s=2 \]
With no output cropping, the output size is:\[ n_{\text{out}} = (n_{\text{in}}-1)s+f \]
Therefore:\[ n_{\text{out}} = (2-1)2+3 = 5 \]
The raw output is consequently \(5\times5\).
Contribution from the Upper-Left Input
The upper-left input value is \(2\). Multiplying the kernel by \(2\) gives:\[ 2K= \begin{bmatrix} 0 & 2 & 0\\ 2 & 4 & 2\\ 0 & 2 & 0 \end{bmatrix} \]
This contribution is placed at the upper-left output position.
Contribution from the Upper-Right Input
The upper-right input value is \(1\):\[ 1K= \begin{bmatrix} 0 & 1 & 0\\ 1 & 2 & 1\\ 0 & 1 & 0 \end{bmatrix} \]
Because the stride is two, this contribution begins two output positions to the right.
Where it overlaps the first contribution, the values are added.
Contribution from the Lower-Left Input
The lower-left value is \(3\):\[ 3K= \begin{bmatrix} 0 & 3 & 0\\ 3 & 6 & 3\\ 0 & 3 & 0 \end{bmatrix} \]
It is placed two positions below the first contribution.
Contribution from the Lower-Right Input
The lower-right value is \(2\), producing another copy of:\[ 2K= \begin{bmatrix} 0 & 2 & 0\\ 2 & 4 & 2\\ 0 & 2 & 0 \end{bmatrix} \]
It is placed two positions down and two positions to the right.
Final Accumulation
Adding all four contributions gives:\[ Y= \begin{bmatrix} 0 & 2 & 0 & 1 & 0\\ 2 & 4 & 3 & 2 & 1\\ 0 & 5 & 0 & 4 & 0\\ 3 & 6 & 5 & 4 & 2\\ 0 & 3 & 0 & 2 & 0 \end{bmatrix} \]
The central locations receive contributions from multiple input values, so their values are sums rather than replacements.
Overlap Uses Addition
Overlap accumulation is an essential part of transposed convolution.
Suppose one input value contributes \(a\) to an output location and another contributes \(b\) to the same location. The final output is:\[ y=a+b \]
The second contribution does not overwrite the first.
This is analogous to the backward-input calculation of an ordinary convolution: multiple paths can contribute to the same input-gradient position, so those contributions must be accumulated.
Output-Size Formula
For one spatial dimension, a common transposed-convolution output formula is:\[ n_{\text{out}} = (n_{\text{in}}-1)s – 2p + d(f-1) + o + 1 \]
where:
- \(n_{\text{in}}\) is the input size.
- \(s\) is the stride.
- \(p\) is the padding parameter.
- \(f\) is the kernel size.
- \(d\) is the dilation.
- \(o\) is the output padding.
- \(n_{\text{out}}\) is the output size.
With dilation one, this becomes:\[ n_{\text{out}} = (n_{\text{in}}-1)s – 2p + f + o \]
The same formula is applied independently to height and width.
Correctly Producing a \(4\times4\) Output
Suppose:\[ n_{\text{in}}=2,\qquad f=3,\qquad s=2,\qquad p=1 \]
Without output padding:\[ n_{\text{out}} = (2-1)2-2(1)+3 = 3 \]
Therefore, these settings alone produce a \(3\times3\) output under the standard formula—not \(4\times4\).
To obtain \(4\times4\), use output padding:\[ o=1 \]
Then:\[ n_{\text{out}} = (2-1)2-2(1)+3+1 = 4 \]
Thus, one valid configuration is:\[ f=3,\qquad s=2,\qquad p=1,\qquad o=1 \]
Another common way to double the resolution is:\[ f=2,\qquad s=2,\qquad p=0,\qquad o=0 \]
because:\[ n_{\text{out}} = (2-1)2+2 = 4 \]
When describing a \(2\times2\) to \(4\times4\) transposed convolution, the kernel, stride, padding, and output-padding conventions must all be stated.
Some frameworks permit the target output shape to be supplied directly, but the requested shape must still be compatible with the operation.
Why Output Padding Exists
For an ordinary strided convolution, several input sizes can produce the same output size because stride skips positions.
For example, an ordinary stride-two convolution may map different input widths to the same smaller width after rounding. When reversing the spatial shape calculation, the desired larger size can therefore be ambiguous.
Output padding resolves this ambiguity.
Despite its name, output padding does not normally add a border of literal zeros to the completed tensor. It adjusts the calculated output shape, typically on the bottom or right side.
For each dimension:\[ 0\leq o<s \]
is a common requirement.
The Matrix Interpretation
An ordinary convolution is a linear transformation. After flattening its tensors, it can be represented as:\[ \mathbf{y}=C\mathbf{x} \]
where \(C\) is a sparse matrix constructed from the convolutional kernel.
A transposed convolution applies:\[ \mathbf{z}=C^{T}\mathbf{y} \]
This is the source of its name.
It is the transpose of the linear operator representing convolution—not necessarily the inverse:\[ C^{T}\neq C^{-1} \]
in general.
This interpretation also explains why contributions overlap and add. Multiplication by \(C^T\) accumulates values from every connected input position.
The Zero-Insertion Interpretation
For stride greater than one, transposed convolution can also be understood conceptually as:
- Insert zeros between input values.
- Pad or crop according to the chosen convention.
- Apply a stride-one convolution-like operation.
- Produce the expanded output.
For a one-dimensional input:\[ (a,b,c) \]
a stride-two expansion conceptually creates:\[ (a,0,b,0,c) \]
A learned kernel is then applied to this expanded representation.
This interpretation is useful, but an efficient implementation does not necessarily construct the zero-filled tensor explicitly.
Transposed Convolution Over Multiple Channels
Real neural networks operate on multi-channel tensors.
Suppose the input is:\[ X \in \mathbb{R}^{H_{\text{in}}\times W_{\text{in}}\times C_{\text{in}}} \]
and the desired output is:\[ Y \in \mathbb{R}^{H_{\text{out}}\times W_{\text{out}}\times C_{\text{out}}} \]
The transposed-convolution kernel connects all input channels to all output channels.
Ignoring framework-specific storage order, the logical kernel dimensions are:\[ f_h\times f_w\times C_{\text{in}}\times C_{\text{out}} \]
Every input-channel value contributes through learned weights to the output channels.
Different libraries store the channel dimensions in different orders, so the required weight-tensor layout should be checked carefully.
Learnable Parameters
Like an ordinary convolution, a transposed-convolution layer contains trainable filters and usually bias parameters.
If the kernel is \(f_h\times f_w\), with \(C_{\text{in}}\) input channels and \(C_{\text{out}}\) output channels, the parameter count is:\[ f_h f_w C_{\text{in}} C_{\text{out}} + C_{\text{out}} \]
The parameters are optimized through backpropagation.
The layer can therefore learn how to transform low-resolution features into a useful higher-resolution representation rather than relying on a fixed interpolation rule.
Transposed Convolution Is Not Ordinary Deconvolution
Transposed convolution is sometimes informally called deconvolution, but that name can be misleading.
True deconvolution usually refers to recovering an original signal by undoing a convolutional process. Transposed convolution generally does not do this.
It:
- Does not recover information lost through pooling
- Does not guarantee reconstruction of the original input
- Does not calculate the inverse of a convolutional kernel
- Applies a learnable linear upsampling transformation
The term transposed convolution is therefore more precise.
Comparison with Interpolation
Another common decoder design uses fixed interpolation followed by an ordinary convolution:\[ X \rightarrow \operatorname{Upsample}(X) \rightarrow \operatorname{Conv} \]
Possible interpolation methods include:
- Nearest-neighbor interpolation
- Bilinear interpolation
- Bicubic interpolation
The two approaches have different properties.
| Transposed convolution | Interpolation followed by convolution |
|---|---|
| Learns the upsampling operation | Uses a fixed resizing operation |
| Combines expansion and filtering | Separates resizing from feature learning |
| Can produce uneven overlap artifacts | Often produces smoother spatial expansion |
| Output shape depends on several parameters | Target size is usually explicit |
Neither approach is universally superior. Both are widely used in segmentation decoders.
Checkerboard Artifacts
Transposed convolutions can create checkerboard-like patterns when some output positions receive more kernel contributions than others.
This problem is especially likely when:\[ f \bmod s \neq 0 \]
For example, a kernel size of three with stride two creates uneven overlap patterns.
Possible ways to reduce these artifacts include:
- Use a kernel size divisible by the stride.
- Use bilinear or nearest-neighbor upsampling followed by convolution.
- Apply additional smoothing convolutions.
- Choose initialization and padding carefully.
For a doubling operation, a \(2\times2\) kernel with stride two has a more uniform overlap pattern than a \(3\times3\) kernel with stride two.
Role in U-Net
In a U-Net decoder, a transposed convolution commonly performs two tasks:
- Increase height and width.
- Reduce or transform the channel dimension.
For example:\[ 32\times32\times512 \]
may become:\[ 64\times64\times256 \]
The result is then concatenated with the corresponding encoder feature map:\[ E \in \mathbb{R}^{64\times64\times256} \]
giving:\[ \operatorname{Concat}(U,E) \in \mathbb{R}^{64\times64\times512} \]
Ordinary convolutional layers refine this combined representation before the next upsampling stage.
The transposed convolution restores spatial scale, while the skip connection restores high-resolution detail lost during encoding.
Common Mistakes
Assuming Stride Two Always Doubles the Size
The output also depends on kernel size, padding, dilation, and output padding:\[ n_{\text{out}} = (n_{\text{in}}-1)s-2p+d(f-1)+o+1 \]
Overwriting Overlapping Contributions
Overlapping kernel contributions must be added.
Treating Output Padding as a Zero Border
Output padding primarily resolves output-shape ambiguity. It does not ordinarily mean attaching a visible border of zeros after the calculation.
Calling the Operation an Exact Inverse
A transposed convolution applies the transpose of a convolution’s linear map. It does not generally invert the original convolution.
Ignoring Spatial Alignment
In architectures with skip connections, the upsampled tensor must match the corresponding encoder tensor’s height and width before concatenation.
Key Takeaway
A transposed convolution expands a feature map by distributing every input value through a learned kernel onto a larger output canvas:\[ \text{input value} \times \text{learned kernel} \longrightarrow \text{output contribution} \]
Overlapping contributions are added, and the output size is determined by:\[ n_{\text{out}} = (n_{\text{in}}-1)s – 2p + d(f-1) + o + 1 \]
This operation provides learnable upsampling for segmentation decoders such as U-Net. It is the transpose of a convolutional linear operator, not an exact inverse of convolution.