Style Loss and Gram Matrices in Neural Style Transfer

ByJu Yeon Eum

Neural style transfer represents an image’s visual style using correlations between feature channels in a pretrained convolutional network.

The central idea is:

Two images have a similar style when their learned visual features tend to occur together in similar ways.

These feature relationships are summarized using a Gram matrix, also called a style matrix. The style loss measures the difference between the Gram matrices of the style image and the generated image.

The Three Images

Neural style transfer uses:

  • \(C\): the content image
  • \(S\): the style image
  • \(G\): the generated image

The objective is to make \(G\):

  • Semantically similar to \(C\)
  • Stylistically similar to \(S\)

A fixed pretrained convolutional network extracts representations from all three images.

Activations at One Layer

Choose convolutional layer \(\ell\). For image \(X\), its activation tensor is:\[ A^{[\ell]}(X) \in \mathbb{R}^{H_\ell\times W_\ell\times C_\ell} \]

where:

  • \(H_\ell\) is the feature-map height.
  • \(W_\ell\) is the feature-map width.
  • \(C_\ell\) is the number of channels.

An individual activation is:\[ a_{i,j,k}^{[\ell]}(X) \]

where:

  • \(i\) indexes height.
  • \(j\) indexes width.
  • \(k\) indexes channels.

Each channel is associated with a learned visual feature.

Interpreting Feature Channels

Suppose one channel responds to vertical textures, while another responds to orange-colored regions.

If these channels activate strongly at the same spatial positions, the image frequently contains orange regions with vertical texture.

If they rarely activate together, those features tend not to co-occur.

The same principle applies across hundreds of channels representing patterns such as:

  • Colors
  • Edges
  • Textures
  • Curves
  • Repeated structures
  • Object fragments

The pattern of co-activation across channels provides a statistical description of the image’s appearance.

Reshaping the Activation Tensor

To calculate all channel relationships efficiently, flatten the spatial dimensions.

Define:\[ M_\ell=H_\ell W_\ell \]

Then reshape:\[ A^{[\ell]}(X) \in \mathbb{R}^{H_\ell\times W_\ell\times C_\ell} \]

into:\[ F^{[\ell]}(X) \in \mathbb{R}^{C_\ell\times M_\ell} \]

Each row contains the activations of one channel across all spatial positions:\[ F_{k,:}^{[\ell]}(X) \]

The particular memory layout must be handled consistently, but the conceptual organization is:

  • Rows: channels
  • Columns: spatial positions

The Gram Matrix

The Gram matrix is:\[ \Gamma^{[\ell]}(X) = F^{[\ell]}(X) F^{[\ell]}(X)^T \]

Its shape is:\[ \Gamma^{[\ell]}(X) \in \mathbb{R}^{C_\ell\times C_\ell} \]

Each element is:\[ \Gamma_{kk’}^{[\ell]}(X) = \sum_{m=1}^{M_\ell} F_{k,m}^{[\ell]}(X) F_{k’,m}^{[\ell]}(X) \]

Using the original spatial indices:\[ \Gamma_{kk’}^{[\ell]}(X) = \sum_{i=1}^{H_\ell} \sum_{j=1}^{W_\ell} a_{i,j,k}^{[\ell]}(X) a_{i,j,k’}^{[\ell]}(X) \]

This value measures the aggregated co-activation of channels \(k\) and \(k’\).

Matrix Dimensions

If:\[ F^{[\ell]} \in \mathbb{R}^{C_\ell\times M_\ell} \]

then:\[ F^{[\ell]T} \in \mathbb{R}^{M_\ell\times C_\ell} \]

Therefore:\[ F^{[\ell]}F^{[\ell]T} \in \mathbb{R}^{C_\ell\times C_\ell} \]

The Gram matrix compares every channel with every other channel.

A frequent implementation mistake is multiplying in the opposite order:\[ F^{[\ell]T}F^{[\ell]} \]

which produces an \(M_\ell\times M_\ell\) spatial-similarity matrix rather than the desired channel-style matrix.

What a Gram-Matrix Element Means

For one pair of channels:\[ \Gamma_{kk’}^{[\ell]} = \sum_m F_{k,m}^{[\ell]}F_{k’,m}^{[\ell]} \]

If both channels tend to have large activations at the same locations, then:\[ \Gamma_{kk’}^{[\ell]} \]

will be large.

If their joint activations are generally small, the value will be smaller.

The diagonal entries are:\[ \Gamma_{kk}^{[\ell]} = \sum_m \left( F_{k,m}^{[\ell]} \right)^2 \]

These measure the total activation energy of each feature channel.

Therefore, the Gram matrix captures both:

  • The prevalence of individual features
  • The co-occurrence of pairs of features

Gram Matrices Are Symmetric

Because:\[ \Gamma^{[\ell]} = FF^T \]

the Gram matrix satisfies:\[ \Gamma_{kk’}^{[\ell]} = \Gamma_{k’k}^{[\ell]} \]

It is also positive semidefinite:\[ v^T\Gamma^{[\ell]}v\geq0 \]

for any vector \(v\).

These properties follow directly from its matrix construction.

“Correlation” Is an Intuition

The Gram matrix is often described as measuring correlations between feature channels. This is useful intuition, but it is not a centered Pearson correlation matrix.

A conventional covariance calculation subtracts channel means:\[ \operatorname{Cov}(k,k’) = \frac{1}{M_\ell} \sum_m \left( F_{k,m}-\mu_k \right) \left( F_{k’,m}-\mu_{k’} \right) \]

The Gram matrix does not subtract means:\[ \Gamma_{kk’} = \sum_m F_{k,m}F_{k’,m} \]

It is more precisely an uncentered second-order feature statistic.

The Gram matrix measures uncentered feature co-activation, not ordinary normalized statistical correlation.

Why Gram Matrices Represent Style

The Gram matrix sums over spatial positions. It records which features occur together but discards much of the information about exactly where they occur.

For example, it may preserve that:

  • Blue features frequently accompany curved edges
  • Orange tones co-occur with repeated short strokes
  • Fine diagonal textures appear throughout the image

It does not strongly preserve the exact coordinates of those patterns.

This makes it suitable for style:

  • Texture and color statistics are retained.
  • Exact scene arrangement is weakened.
  • Local patterns can be reproduced at new positions.

Content representations preserve more spatial structure, while Gram matrices deliberately discard much of it.

Style Gram Matrix

For the style image \(S\), calculate:\[ \Gamma_S^{[\ell]} = \Gamma^{[\ell]}(S) \]

For the generated image \(G\), calculate:\[ \Gamma_G^{[\ell]} = \Gamma^{[\ell]}(G) \]

The goal is:\[ \Gamma_G^{[\ell]} \approx \Gamma_S^{[\ell]} \]

If the matrices are similar, the generated image has similar feature co-activation statistics at layer \(\ell\).

Style Loss for One Layer

A basic style loss is the squared Frobenius distance:\[ J_{\text{style}}^{[\ell]}(S,G) = \left\| \Gamma_G^{[\ell]} – \Gamma_S^{[\ell]} \right\|_F^2 \]

Expanded:\[ J_{\text{style}}^{[\ell]}(S,G) = \sum_{k=1}^{C_\ell} \sum_{k’=1}^{C_\ell} \left( \Gamma_{G,kk’}^{[\ell]} – \Gamma_{S,kk’}^{[\ell]} \right)^2 \]

Minimizing this loss makes the generated image reproduce the style image’s channel relationships.

Normalized Style Loss

The unnormalized Gram values grow with the number of spatial positions and channels. A common normalized loss is:\[ J_{\text{style}}^{[\ell]}(S,G) = \frac{ \left\| \Gamma_G^{[\ell]} – \Gamma_S^{[\ell]} \right\|_F^2 }{ 4C_\ell^2M_\ell^2 } \]

Since:\[ M_\ell=H_\ell W_\ell \]

this is equivalent to:\[ J_{\text{style}}^{[\ell]}(S,G) = \frac{ \displaystyle \sum_{k,k’} \left( \Gamma_{G,kk’}^{[\ell]} – \Gamma_{S,kk’}^{[\ell]} \right)^2 }{ 4C_\ell^2H_\ell^2W_\ell^2 } \]

It can also be written with the denominator:\[ \left( 2H_\ell W_\ell C_\ell \right)^2 \]

because:\[ \left( 2H_\ell W_\ell C_\ell \right)^2 = 4C_\ell^2H_\ell^2W_\ell^2 \]

Different implementations distribute normalization between the Gram matrix, layer loss, and layer weight. Consistency matters more than using one universal convention.

Computing Style at Multiple Layers

A single layer captures visual statistics at one level of abstraction.

For richer results, choose a collection of style layers:\[ \mathcal{L}_S = \{\ell_1,\ell_2,\ldots,\ell_L\} \]

The total style loss is:\[ J_{\text{style}}(S,G) = \sum_{\ell\in\mathcal{L}_S} \lambda_\ell J_{\text{style}}^{[\ell]}(S,G) \]

where:\[ \lambda_\ell\geq0 \]

controls the contribution of each layer.

The weights are often normalized so that:\[ \sum_{\ell\in\mathcal{L}_S} \lambda_\ell=1 \]

although this is not required.

What Different Style Layers Capture

Early Layers

Early layers respond to:

  • Colors
  • Edges
  • Fine textures
  • Small strokes
  • Local contrast

Matching their Gram matrices transfers fine-scale appearance.

Middle Layers

Middle layers capture:

  • Repeated motifs
  • More structured textures
  • Curves
  • Larger patterns
  • Combinations of edges and colors

Deeper Layers

Deeper layers may capture:

  • Large visual elements
  • Broader spatial motifs
  • More abstract pattern combinations
  • Semantically influenced structure

Using several layers lets the generated image reproduce style at multiple scales.

Content Loss

Choose a content layer \(\ell_c\). Let:\[ A_C^{[\ell_c]} = \phi^{[\ell_c]}(C) \]

and:\[ A_G^{[\ell_c]} = \phi^{[\ell_c]}(G) \]

A common content loss is:\[ J_{\text{content}}(C,G) = \frac{1}{2} \left\| A_G^{[\ell_c]} – A_C^{[\ell_c]} \right\|_F^2 \]

Unlike the style loss, the content loss compares activation tensors at corresponding spatial locations. It therefore preserves more of the scene’s structure.

Total Objective

The complete objective combines content and style:\[ J(G) = \alpha J_{\text{content}}(C,G) + \beta J_{\text{style}}(S,G) \]

where:

  • \(\alpha\) controls content preservation.
  • \(\beta\) controls style strength.

An optional smoothness term can be added:\[ J(G) = \alpha J_{\text{content}} + \beta J_{\text{style}} + \gamma J_{\text{TV}} \]

where \(J_{\text{TV}}\) is a total-variation regularizer.

Optimizing the Generated Image

The pretrained feature extractor remains fixed. Optimization updates only the generated image:\[ G \leftarrow G – \eta \frac{\partial J(G)}{\partial G} \]

The gradient of the style loss passes through:\[ G \rightarrow F_G^{[\ell]} \rightarrow \Gamma_G^{[\ell]} \rightarrow J_{\text{style}} \]

Automatic differentiation calculates these gradients through all selected layers.

Efficient Gram-Matrix Calculation

Suppose an activation tensor is stored as:\[ A\in\mathbb{R}^{B\times C\times H\times W} \]

for a batch-first, channels-first framework.

It can be reshaped into:\[ F\in\mathbb{R}^{B\times C\times HW} \]

The batch of Gram matrices is:\[ \Gamma = FF^T \]

where the multiplication is performed independently for every batch item.

A PyTorch-style implementation is:

import torch


def gram_matrix(features):
    batch_size, channels, height, width = features.shape

    flattened = features.reshape(
        batch_size,
        channels,
        height * width
    )

    gram = torch.bmm(
        flattened,
        flattened.transpose(1, 2)
    )

    return gram / (channels * height * width)

This implementation normalizes the Gram matrix directly. If the Gram matrix is normalized here, the later style-loss formula should not apply the same normalization a second time unless that behavior is intentional.

Style-Loss Implementation

import torch.nn.functional as F


def style_loss(generated_features, target_gram):
    generated_gram = gram_matrix(generated_features)

    return F.mse_loss(
        generated_gram,
        target_gram
    )

For multiple style layers:

def multi_layer_style_loss(
    generated_activations,
    target_grams,
    layer_weights
):
    total = 0.0

    for layer_name, weight in layer_weights.items():
        total = total + weight * style_loss(
            generated_activations[layer_name],
            target_grams[layer_name]
        )

    return total

The target style Gram matrices can be computed once because the style image and feature extractor remain fixed.

Precomputing Fixed Targets

Before optimizing \(G\), calculate and detach:\[ A_C^{[\ell_c]} \]

and:\[ \Gamma_S^{[\ell]} \quad \text{for each selected style layer} \]

These targets do not change during optimization.

Only the generated image’s activations and Gram matrices must be recomputed at every step.

This reduces unnecessary computation and prevents accidental gradient tracking through the fixed target images.

Interpreting the Diagonal and Off-Diagonal Terms

The diagonal element:\[ \Gamma_{kk} \]

measures the overall strength of channel \(k\).

The off-diagonal element:\[ \Gamma_{kk’} \]

measures how strongly channels \(k\) and \(k’\) activate together.

Matching only diagonal values would approximately match individual feature energies. Matching the complete Gram matrix also matches pairwise relationships, creating a richer style representation.

Spatial Information Is Reduced, Not Completely Irrelevant

The Gram matrix sums over spatial positions, so it is invariant to a common permutation of those positions at that layer:\[ F \rightarrow FP \]

where \(P\) is a permutation matrix. Then:\[ (FP)(FP)^T = FPP^TF^T = FF^T \]

This shows why the Gram matrix does not encode the exact location of each activation.

However, the generated image is still constrained by:

  • Convolutional receptive fields
  • Content loss
  • Multiple style layers
  • Natural image structure
  • Optimization dynamics

The final result is therefore not spatially random.

Choosing Style-Layer Weights

The weights \(\lambda_\ell\) determine the scale of transferred appearance.

More weight on shallow layers tends to emphasize:

  • Fine texture
  • Color
  • Small strokes

More weight on deeper layers tends to emphasize:

  • Larger patterns
  • Structured motifs
  • Broader feature arrangements

A balanced starting point is to distribute weight across several early and middle layers and then adjust based on the generated result.

Because layer losses can have different numerical scales, equal weights do not necessarily imply equal influence.

Choosing Content and Style Weights

The balance between:\[ \alpha \]

and:\[ \beta \]

controls the major visual tradeoff.

If:\[ \beta\gg\alpha \]

the output becomes more stylized but may distort content.

If:\[ \alpha\gg\beta \]

the content remains clear but the style may be weak.

Because normalization affects loss magnitudes, values from one implementation may not transfer directly to another.

Common Mistakes

Calling It an Ordinary Correlation Matrix

The standard Gram matrix does not subtract means or divide by standard deviations. It is an uncentered second-order statistic.

Using the Wrong Matrix Multiplication

For channel correlations, use:\[ FF^T \]

not:\[ F^TF \]

Mixing Channel and Spatial Dimensions

The activation tensor must be reshaped so that one dimension indexes channels and the other indexes flattened spatial positions.

Comparing Raw Features for Style

Directly matching:\[ A_G^{[\ell]} \approx A_S^{[\ell]} \]

preserves the style image’s spatial arrangement too strongly. Gram matrices intentionally weaken exact spatial correspondence.

Double-Normalizing Accidentally

If the Gram matrix is normalized when constructed, account for that when defining the layer loss.

Optimizing the Network Weights

In classical optimization-based style transfer, the feature extractor is frozen. The generated pixels are optimized.

Using Only One Layer Without Checking the Result

One layer may capture style at an overly narrow scale. Multiple layers usually produce richer output.

Limitations of Gram-Matrix Style

Gram matrices capture useful texture and co-activation statistics, but they do not represent every aspect of artistic style.

They may not preserve:

  • Exact brushstroke placement
  • Global composition
  • Semantic symbolism
  • Deliberate object deformation
  • Long-range spatial relationships
  • Artist-specific creative intent

Two perceptually different images can have similar Gram matrices because spatial organization is largely discarded.

Despite these limitations, Gram matrices provide a remarkably effective differentiable representation of texture-like appearance.

Key Takeaway

At layer \(\ell\), reshape the activation tensor into:\[ F^{[\ell]} \in \mathbb{R}^{C_\ell\times H_\ell W_\ell} \]

and calculate the Gram matrix:\[ \Gamma^{[\ell]} = F^{[\ell]}F^{[\ell]T} \]

The style loss compares the generated and style Gram matrices:\[ J_{\text{style}}^{[\ell]}(S,G) = \frac{ \left\| \Gamma_G^{[\ell]} – \Gamma_S^{[\ell]} \right\|_F^2 }{ 4C_\ell^2(H_\ell W_\ell)^2 } \]

Using several layers captures visual statistics at multiple scales:\[ J_{\text{style}}(S,G) = \sum_\ell \lambda_\ell J_{\text{style}}^{[\ell]}(S,G) \]

Combined with content loss, this objective allows the generated image to preserve the content image’s semantic structure while adopting the style image’s feature statistics.

Similar Posts

Leave a Reply