Convolutional Implementation of Sliding-Window Object Detection
Traditional sliding-window detection repeatedly crops regions from an image and processes every crop independently. This performs a large amount of duplicated computation because neighboring windows overlap substantially.
A convolutional implementation eliminates most of this duplication. It processes the complete image in a single forward pass and produces predictions for many overlapping windows simultaneously.
The key step is converting the network’s fully connected layers into convolutional layers.
A Conventional Window Classifier
Suppose an object classifier accepts an input image with shape:\[ 14 \times 14 \times 3 \]
Its architecture might be:
- A convolution with sixteen \(5 \times 5\) filters
- A \(2 \times 2\) max-pooling operation
- A fully connected layer with 400 units
- Another fully connected layer with 400 units
- A four-class softmax output
The corresponding activation shapes are:\[ 14 \times 14 \times 3 \]\[ \downarrow \text{ Conv: } f=5,\; s=1,\; p=0 \]\[ 10 \times 10 \times 16 \]\[ \downarrow \text{ MaxPool: } f=2,\; s=2 \]\[ 5 \times 5 \times 16 \]
The network then flattens the last volume and feeds it into the fully connected layers.
The four output classes could be:
- Pedestrian
- Car
- Motorcycle
- Background
Converting a Fully Connected Layer into a Convolution
The first fully connected layer receives:\[ 5 \times 5 \times 16 = 400 \]
input activations and produces 400 outputs.
Each output unit is connected to every value in the \(5 \times 5 \times 16\) input volume. The same calculation can be implemented using one convolutional filter with shape:\[ 5 \times 5 \times 16 \]
for each output unit.
Because the layer contains 400 output units, it can be replaced by 400 filters:\[ W^{[1]} \in \mathbb{R}^{5 \times 5 \times 16 \times 400} \]
Applying these filters to the \(5 \times 5 \times 16\) input produces:\[ 1 \times 1 \times 400 \]
Therefore:\[ \text{FC}(400) \quad\Longleftrightarrow\quad \text{Conv}(400,\;5\times5) \]
For this specific input size, the two layers perform the same mathematical operation.
Converting the Remaining Fully Connected Layers
The next fully connected layer maps 400 values to another 400 values.
Because its input is now represented as:\[ 1 \times 1 \times 400 \]
it can be replaced by 400 filters with shape:\[ 1 \times 1 \times 400 \]
Its output remains:\[ 1 \times 1 \times 400 \]
The final fully connected classification layer maps these 400 values to four class scores. It can therefore be implemented using four filters with shape:\[ 1 \times 1 \times 400 \]
The output becomes:\[ 1 \times 1 \times 4 \]
After conversion, the entire network is convolutional:\[ 14 \times 14 \times 3 \rightarrow 10 \times 10 \times 16 \rightarrow 5 \times 5 \times 16 \rightarrow 1 \times 1 \times 400 \rightarrow 1 \times 1 \times 400 \rightarrow 1 \times 1 \times 4 \]
Why the Conversion Is Equivalent
A fully connected unit calculates:\[ z_k = \sum_{i=1}^{5} \sum_{j=1}^{5} \sum_{c=1}^{16} W_{i,j,c,k}a_{i,j,c} + b_k \]
A convolutional filter covering the complete \(5 \times 5 \times 16\) input calculates exactly the same expression.
The difference is how the operation behaves when the spatial input becomes larger.
A fully connected layer expects a fixed-size vector. A convolutional layer can slide across a larger feature map, applying the same learned operation at every valid spatial position.
A fully connected layer can be viewed as a convolution whose filter covers the complete spatial extent of its input.
Applying the Network to a Larger Image
The original network was trained on \(14 \times 14 \times 3\) image patches. Now suppose the test image has shape:\[ 16 \times 16 \times 3 \]
A traditional sliding-window method would extract four \(14 \times 14\) windows using a stride of two:
- Upper-left window
- Upper-right window
- Lower-left window
- Lower-right window
It would then evaluate the network four separate times.
Because these windows overlap heavily, most low-level convolutional calculations would be repeated.
Processing the Complete Image Once
Instead, feed the entire \(16 \times 16 \times 3\) image through the fully convolutional network.
The first convolution produces:\[ 16-5+1=12 \]
so its output has shape:\[ 12 \times 12 \times 16 \]
The \(2 \times 2\) max-pooling layer with stride two produces:\[ 6 \times 6 \times 16 \]
The converted fully connected layer now applies its \(5 \times 5\) filters at every valid position:\[ 6-5+1=2 \]
Its output therefore becomes:\[ 2 \times 2 \times 400 \]
The two \(1 \times 1\) convolutional layers preserve the spatial dimensions:\[ 2 \times 2 \times 400 \rightarrow 2 \times 2 \times 400 \rightarrow 2 \times 2 \times 4 \]
The final result is:\[ 2 \times 2 \times 4 \]
Each of the four spatial locations corresponds to one of the four \(14 \times 14\) sliding windows.
Equivalence to Four Independent Evaluations
The four output positions represent:
| Output position | Corresponding input window |
|---|---|
| Upper left | Upper-left \(14 \times 14\) crop |
| Upper right | Upper-right \(14 \times 14\) crop |
| Lower left | Lower-left \(14 \times 14\) crop |
| Lower right | Lower-right \(14 \times 14\) crop |
Thus, one forward pass computes the same four classifications that would otherwise require four separate network evaluations.
The convolutional version is faster because the early feature calculations for overlapping regions are shared.
A Larger Example
Suppose the input is:\[ 28 \times 28 \times 3 \]
The same network produces the following shapes.
After the \(5 \times 5\) convolution:\[ 28-5+1=24 \]\[ 24 \times 24 \times 16 \]
After \(2 \times 2\) max pooling with stride two:\[ 12 \times 12 \times 16 \]
After the converted \(5 \times 5\) fully connected layer:\[ 12-5+1=8 \]\[ 8 \times 8 \times 400 \]
After the two \(1 \times 1\) convolutions:\[ 8 \times 8 \times 400 \rightarrow 8 \times 8 \times 4 \]
The final output is therefore:\[ 8 \times 8 \times 4 \]
This represents class predictions for:\[ 8 \times 8=64 \]
overlapping windows, all calculated in a single forward pass.
Why the Effective Window Stride Is Two
The prediction grid advances by two pixels in the original image because the network contains a max-pooling layer with stride two.
More generally, the effective stride of an output prediction relative to the input is the product of the strides in all preceding convolutional and pooling layers:\[ s_{\text{effective}} = \prod_{\ell} s^{[\ell]} \]
In this example, the convolution has stride one and the pooling layer has stride two:\[ s_{\text{effective}} = 1 \times 2 = 2 \]
Adjacent output locations therefore correspond to input windows shifted by two pixels.
If the network contained two stride-two pooling layers, its effective stride would be:\[ 2 \times 2=4 \]
This would make inference faster but produce a coarser localization grid.
Receptive Fields and Sliding Windows
Each output location depends on a particular region of the original image. That region is the output unit’s receptive field.
In this architecture, each final spatial position has a receptive field corresponding to one \(14 \times 14\) input window. Moving to an adjacent output location shifts the receptive field by the network’s effective stride.
This provides a more general interpretation:
Convolutional sliding windows classify the receptive field associated with every output position.
The network does not explicitly crop each window. Its layered convolutional operations generate the equivalent results through shared feature maps.
Computational Advantage
Let \(N\) be the number of candidate windows.
The naive approach performs approximately:\[ N \times \text{cost of one complete network evaluation} \]
The convolutional approach computes shared feature maps once and evaluates the later filters densely over those maps.
The exact speedup depends on:
- The amount of overlap between windows
- Network depth
- Feature-map dimensions
- Effective stride
- Hardware and implementation efficiency
The improvement is especially substantial when neighboring windows overlap heavily.
The Remaining Localization Problem
Although convolutional sliding windows improve computational efficiency, the possible output locations are still tied to a discrete grid.
The model effectively chooses among predefined receptive fields. Consequently:
- The box center is restricted by the output stride.
- Box dimensions may be fixed or selected from a limited collection.
- The best available window may not align precisely with the object.
- Non-square or unusually shaped objects may be localized poorly.
Using a finer output stride improves spatial precision, but it increases computation and still does not fully solve the fixed-box problem.
Direct bounding-box regression addresses this limitation by allowing the network to predict continuous coordinates:\[ (b_x,b_y,b_h,b_w) \]
rather than merely selecting a fixed window.
Relationship to Modern Dense Detectors
The convolutional sliding-window technique introduced several principles that remain central to modern object detection:
- Shared computation over the complete image
- Dense spatial predictions
- Fully convolutional classification
- Receptive fields corresponding to candidate object regions
- One-pass generation of predictions across many locations
Modern one-stage detectors extend these ideas by predicting objectness, classes, and adjustable bounding boxes at every spatial location, often across multiple feature-map scales.
Advantages and Limitations
| Aspect | Convolutional sliding windows |
|---|---|
| Number of forward passes | One per image or scale |
| Feature sharing | Yes |
| Classification efficiency | Much better than independent crops |
| Spatial predictions | Dense output grid |
| Bounding-box flexibility | Limited without regression |
| Localization precision | Constrained by effective stride |
| Multiple object sizes | Requires multiple scales or additional mechanisms |
Key Takeaway
Fully connected layers can be converted into convolutional layers:\[ \text{FC over }H\times W\times C \quad\Longleftrightarrow\quad \text{Conv with }H\times W\times C\text{ filters} \]
This conversion allows a window classifier to process a larger image and produce predictions for many overlapping windows in one forward pass. The method shares convolutional computation across neighboring regions, making sliding-window detection substantially more efficient. Its main remaining weakness is that localization is still constrained by a discrete prediction grid and fixed receptive fields, motivating direct bounding-box regression.