Visualizing What Deep Convolutional Networks Learn

A convolutional neural network does not receive explicit instructions to detect edges, textures, object parts, or complete objects. These internal features emerge during training because they help minimize the model’s objective.

A useful way to study these learned representations is to find the image regions that produce the strongest activation in selected hidden units.

These visualizations often reveal a hierarchy:\[ \text{edges and colors} \rightarrow \text{textures and patterns} \rightarrow \text{object parts} \rightarrow \text{semantic concepts} \]

This hierarchy also helps explain why neural style transfer can separate the visual content of an image from its style.

Hidden Activations in a ConvNet

Suppose a trained convolutional network receives an image:\[ x\in\mathbb{R}^{H\times W\times3} \]

At convolutional layer \(\ell\), it produces an activation tensor:\[ A^{[\ell]}(x) \in \mathbb{R}^{H_\ell\times W_\ell\times C_\ell} \]

where:

• \(H_\ell\) is the feature-map height.
• \(W_\ell\) is the feature-map width.
• \(C_\ell\) is the number of channels.

An individual activation can be written as:\[ a_{i,j,k}^{[\ell]}(x) \]

This is the response at:

• Spatial row \(i\)
• Spatial column \(j\)
• Channel \(k\)

Each channel behaves like a learned feature detector applied across many spatial positions.

Units, Channels, and Feature Maps

The word unit can refer to one scalar activation:\[ a_{i,j,k}^{[\ell]} \]

A channel or feature map contains activations for one learned filter across all positions:\[ A_{:,:,k}^{[\ell]} \]

These are different objects:

• A unit responds to a feature at one location.
• A channel responds to a similar feature across many locations.

When visualizing a feature detector, researchers may rank images by:

• One spatial unit
• The maximum activation anywhere in one channel
• The average activation across a channel
• Another aggregated channel score

The interpretation depends on the score being visualized.

Dataset-Based Activation Visualization

One simple visualization technique uses real images from a dataset.

For a selected channel \(k\) in layer \(\ell\), define a score such as:\[ s_k^{[\ell]}(x) = \max_{i,j} a_{i,j,k}^{[\ell]}(x) \]

Then:

1. Run many images through the trained network.
2. Record the activation score for each image.
3. Rank the images by that score.
4. Select the highest-scoring examples.
5. Extract the corresponding receptive-field patches.

If nine image patches produce the strongest responses, those patches provide evidence about what the channel prefers.

Why Visualize Image Patches?

An early hidden unit does not see the complete image. It responds only to a local region called its receptive field.

If an early unit responds strongly to an edge in the upper-left portion of its receptive field, displaying the complete source image may obscure that relationship.

Instead, visualize the input patch that could influence the selected activation.

This often makes the learned feature easier to interpret:

• Diagonal boundary
• Vertical edge
• Color transition
• Repeated line pattern
• Curved contour

Receptive Fields

The receptive field of a hidden unit is the region of the original image that can influence that unit.

Early layers have small receptive fields. Deeper layers have larger receptive fields because each layer combines information from previous layers.

For a layer with kernel size \(f_\ell\) and stride \(s_\ell\), define:

• \(r_\ell\): receptive-field size
• \(j_\ell\): spacing between adjacent receptive-field centers in input coordinates

Starting with:\[ r_0=1,\qquad j_0=1 \]

the recursive relationships are:\[ j_\ell = j_{\ell-1}s_\ell \]\[ r_\ell = r_{\ell-1} + (f_\ell-1)j_{\ell-1} \]

Pooling and strided convolutions increase receptive fields particularly quickly.

Receptive-Field Example

Consider three \(3\times3\) convolutions with stride one and no dilation.

The first layer has:\[ r_1 = 1+(3-1)(1) = 3 \]

The second layer has:\[ r_2 = 3+(3-1)(1) = 5 \]

The third layer has:\[ r_3 = 5+(3-1)(1) = 7 \]

Thus, a third-layer unit can depend on a \(7\times7\) area of the original image.

If a stride-two operation is introduced, the receptive field grows more rapidly because later filters move over input positions with larger spacing.

Early Layers: Edges and Color Contrasts

The first convolutional layer operates close to raw pixel values.

Its filters often respond to simple patterns such as:

• Horizontal edges
• Vertical edges
• Diagonal edges
• Bright-to-dark transitions
• Dark-to-bright transitions
• Particular colors
• Color-opponent patterns

A channel might activate for a vertical boundary with green on one side. Another might respond to orange regions or diagonal bright lines.

These features resemble classical edge and color filters, but their numerical values are learned from data rather than manually specified.

Why Early Features Are Simple

An early unit has:

• A small receptive field
• Access only to low-level pixel information
• Too little context to recognize a complete object

It cannot reliably detect an entire dog or car because most of the object lies outside its receptive field.

Instead, it learns small patterns that can be reused throughout the image and combined by deeper layers.

Middle Layers: Textures and Repeated Patterns

Second and middle layers combine multiple early features.

They may respond to:

• Repeated vertical lines
• Grids
• Honeycomb patterns
• Curved contours
• Corners
• Stripes
• Fur-like textures
• Circular fragments
• Repeated dots

A texture detector can be understood as a learned combination of edge detectors at several positions and orientations.

For example, a vertical-stripe feature may require multiple vertical edges arranged with consistent spacing.

Deeper Layers: Object Parts

As the receptive field grows, deeper units can respond to larger and more structured patterns:

• Wheels
• Eyes
• Faces
• Legs
• Text regions
• Animal heads
• Vehicle fronts
• Repeated keyboard keys

These features are more invariant than early edge detectors. A deep channel may respond to a concept across moderate changes in:

• Position
• Scale
• Color
• Lighting
• Background
• Exact shape

This invariance is useful for recognition but can make a channel harder to describe using one simple phrase.

Late Layers: Semantic Concepts

Late convolutional layers may respond strongly to:

• Dogs
• Flowers
• Water
• Text
• Faces
• Keyboards
• Particular object categories or subcategories

A channel described as a “dog detector” may not respond to every dog. It may prefer:

• Particular poses
• Similar head shapes
• Certain textures
• A recurring context
• A combination of dog-like parts

Likewise, a channel that appears to detect water may actually respond to a more general combination of color, reflectivity, and texture.

A visualization suggests what activates a feature; it does not prove that the unit represents one clean human concept.

The Emerging Feature Hierarchy

A simplified hierarchy is:

This is a common pattern, not a rigid law. Actual representations depend on:

• Architecture
• Training objective
• Dataset
• Augmentation
• Initialization
• Regularization
• The selected layer and channel

Some early channels may be difficult to interpret, and some late channels may still encode texture rather than complete objects.

Distributed Representation

A neural network does not necessarily store one concept in one channel.

A concept such as “dog” may be distributed across many channels representing:

• Fur texture
• Ears
• Eyes
• Snout
• Body shape
• Pose
• Background context

Conversely, one channel may respond to several visually related concepts.

Therefore:\[ \text{one channel} \not\equiv \text{one semantic concept} \]

A more accurate interpretation is that concepts are represented by patterns across many channels and spatial locations.

Feature Visualization by Input Optimization

Dataset-based visualization can show only patterns already present in the selected dataset.

Another approach directly optimizes an input image to maximize a chosen activation.

For a channel score \(s_k^{[\ell]}(x)\), solve:\[ x^* = \underset{x}{\operatorname{argmax}} \; s_k^{[\ell]}(x) – \lambda R(x) \]

where:

• \(x\) is the generated image.
• \(R(x)\) is an image regularizer.
• \(\lambda\) controls regularization strength.

Gradient ascent updates the image:\[ x \leftarrow x + \eta \frac{ \partial s_k^{[\ell]}(x) }{ \partial x } \]

where \(\eta\) is the step size.

Why Regularization Is Needed

Without regularization, activation maximization often produces noisy high-frequency patterns that strongly activate the model but are difficult for humans to interpret.

Possible regularizers include:

• Total variation
• Gaussian blur
• Small random translations
• Frequency penalties
• Pixel-value constraints
• Natural-image priors

These encourage smoother and more interpretable visualizations.

Dataset Examples Versus Synthetic Visualization

Using both methods provides stronger evidence than relying on only one.

Projection-Based Visualization

Another family of methods maps an activation back toward input space to estimate which image patterns contributed to it.

Examples include:

• Deconvolution-style visualization
• Guided backpropagation
• Saliency maps
• Gradient-based localization
• Activation maximization

These methods answer slightly different questions.

For example:

• Activation maximization asks what pattern strongly excites a unit.
• A saliency method asks which input pixels influenced a particular prediction.
• Dataset ranking asks which real examples naturally produce high activation.

Their visual outputs should not be interpreted as interchangeable explanations.

Correlation Is Not Causation

Suppose a channel’s strongest examples all contain dogs. This does not necessarily prove that the channel is causally responsible for recognizing dogs.

It may instead detect:

• Fur
• Grass backgrounds
• Curved animal shapes
• A recurring photographic composition
• Another feature correlated with dogs

A stronger investigation could combine:

• Activation visualization
• Feature ablation
• Controlled image editing
• Activation intervention
• Counterfactual tests
• Performance measurement after removing the channel

Visualization is valuable for forming hypotheses, but it is not by itself a complete causal explanation.

Why Deeper Features Become More Semantic

Each layer composes features from the layer before it.

For example:\[ \text{edges} \rightarrow \text{corners} \rightarrow \text{contours} \rightarrow \text{object parts} \rightarrow \text{objects} \]

Mathematically:\[ A^{[\ell]} = g \left( W^{[\ell]}*A^{[\ell-1]}+b^{[\ell]} \right) \]

A channel in layer \(\ell\) can respond to a learned arrangement of features from layer \(\ell-1\).

Repeated composition allows the model to construct increasingly complex representations.

Spatial Resolution and Semantic Abstraction

As depth increases, many networks follow two trends:\[ H_\ell,\;W_\ell\downarrow \]

and:\[ C_\ell\uparrow \]

The spatial feature maps become smaller, but the channel representation becomes richer.

Early layers retain precise spatial detail. Late layers retain more abstract meaning and larger contextual relationships.

This tradeoff is important not only for classification but also for segmentation, detection, and neural style transfer.

Relationship to Transfer Learning

Early CNN features are often useful across many tasks because edges, colors, and textures occur in numerous image domains.

Later features are usually more specialized to the original training data and objective.

This helps explain a common transfer-learning pattern:

• Reuse early and middle layers.
• Replace the original output head.
• Fine-tune later layers for the new task.
• Fine-tune more of the network when more task-specific data is available.

The feature hierarchy makes pretrained convolutional models useful beyond the dataset on which they were initially trained.

Relationship to Neural Style Transfer

The layer hierarchy provides the central intuition behind neural style transfer.

Content Representation

A deeper activation tensor captures higher-level structure:

• Object arrangement
• Broad shapes
• Semantic content
• Spatial organization

A generated image can preserve content by matching its deeper activations to those of a content image.

Style Representation

Earlier and intermediate layers respond strongly to:

• Colors
• Textures
• Brushstroke-like patterns
• Repeated local structures

Style can be represented using correlations among feature channels rather than exact spatial positions.

This is commonly expressed through a Gram matrix:\[ G_{kk’}^{[\ell]} = \sum_{i,j} a_{i,j,k}^{[\ell]} a_{i,j,k’}^{[\ell]} \]

The Gram matrix measures how strongly feature channels activate together across spatial locations.

This distinction motivates neural style transfer:\[ \text{generated image} = \text{deep content structure} + \text{multilayer texture statistics} \]

Practical Visualization Procedure

A simple real-image activation study can follow these steps:

select a trained convolutional network
    ↓
choose a layer and channel
    ↓
run a large image collection through the network
    ↓
calculate a channel score for every image
    ↓
rank images by activation
    ↓
locate the strongest spatial response
    ↓
extract its receptive-field patch
    ↓
inspect the top examples together

select a trained convolutional network
    ↓
choose a layer and channel
    ↓
run a large image collection through the network
    ↓
calculate a channel score for every image
    ↓
rank images by activation
    ↓
locate the strongest spatial response
    ↓
extract its receptive-field patch
    ↓
inspect the top examples together

Displaying several top examples is more informative than showing only one. Repeated visual structure across many examples provides stronger evidence about the feature’s preference.

Common Interpretation Mistakes

Treating Every Channel as a Named Detector

Many channels do not correspond to one human-interpretable concept.

Comparing Unmatched Crop Sizes

Deeper units have larger receptive fields. Displaying all layers using visually identical crop scales can hide this difference.

Ignoring Spatial Position

A channel may respond to a feature anywhere, while one specific unit responds only at one feature-map location.

Assuming the Strongest Patch Is the Only Trigger

A neuron may respond to several distinct visual patterns.

Confusing Visualization Methods

Dataset examples, optimized images, gradients, and projected activations answer different questions.

Assuming Late Layers Ignore Texture

Deep networks can remain strongly sensitive to texture and dataset-specific correlations. The edge-to-object hierarchy is useful intuition, but it is not absolute.

Key Takeaway

A convolutional network often develops a hierarchy of learned visual features:\[ \text{pixels} \rightarrow \text{edges} \rightarrow \text{textures} \rightarrow \text{parts} \rightarrow \text{semantic patterns} \]

This can be studied by finding real image patches that maximize selected activations or by optimizing synthetic inputs directly.

Early layers tend to represent localized color and edge information. Middle layers combine these features into textures and shapes. Deeper layers have larger receptive fields and can respond to object parts or semantic configurations.

These representations are distributed and should be interpreted cautiously, but their hierarchy provides an important foundation for transfer learning, semantic segmentation, and neural style transfer.