Neural Style Transfer: Feature Visualization, Gram Matrices, and Optimization

In classical neural style transfer, a trained CNN defines what to compare between images. Its weights stay fixed while the output image’s pixels change. A content loss favors the structure of one reference image; a style loss favors feature statistics of another. Looking at CNN features helps explain what those losses can preserve and what they leave unconstrained.

What CNN layers learn

One way to inspect a channel is to collect image patches that strongly activate it. Another is activation maximization: start with an image and adjust its pixels to increase a chosen activation. The two approaches answer related questions, but an optimized input may exploit patterns uncommon in real images. Visualizations are evidence about a response, not a unique description of what a channel “means.”

Trained image classifiers often show edge and color responses in early layers and more complex patterns deeper in the network. Larger receptive fields make broader combinations possible, while the training data and objective determine which combinations are useful. A late channel need not correspond to one recognizable object; responses can be distributed across channels or mix several patterns.

Gatys-style transfer uses spatially aligned feature maps as a content descriptor and spatially aggregated channel products as a style descriptor. This is an operational choice of losses, not a complete separation of an image’s content and style. We will keep RGB images in the range [0, 1] and normalize only inside the extractor. Run the Python blocks in order; PyTorch, torchvision, and Pillow are required.

import math
import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision

class FeatureExtractor(nn.Module):
    """Selected activations; inputs are RGB tensors (N, 3, H, W) in [0, 1]."""
    def __init__(self, layers, features=None):
        super().__init__()
        if features is None:
            features = torchvision.models.vgg19(
                weights=torchvision.models.VGG19_Weights.IMAGENET1K_V1).features
        self.layers = set(layers)
        if not self.layers or any(not isinstance(i, int) or i < 0 or i >= len(features)
                                  for i in self.layers):
            raise ValueError("Select valid feature-layer indices")
        self.vgg = features[:max(self.layers) + 1]
        for module in self.vgg.modules():
            if isinstance(module, nn.ReLU):
                module.inplace = False
        self.vgg.eval().requires_grad_(False)
        self.register_buffer("mean", torch.tensor([0.485, 0.456, 0.406])[None, :, None, None])
        self.register_buffer("std", torch.tensor([0.229, 0.224, 0.225])[None, :, None, None])

    def forward(self, x):
        x = (x - self.mean) / self.std
        out = {}
        for i, layer in enumerate(self.vgg):
            x = layer(x)
            if i in self.layers:
                out[i] = x
        return out

This example uses pretrained VGG-19 to make the layer choices explicit. In torchvision’s VGG-19 feature sequence, convolution indices 0, 5, 10, 19, and 28 are conv1_1 through conv5_1; index 21 is conv4_2. We record pre-ReLU outputs. Replacing in-place ReLUs prevents a later operation from silently changing an activation already stored in the dictionary. Post-ReLU features are another valid convention, but targets and generated features must use the same one. Creating the default extractor downloads pretrained weights if they are not cached.

To visualize a channel, minimize its negative mean activation. A total-variation penalty discourages abrupt changes between neighboring pixels, and a small squared-pixel penalty discourages large values. These regularizers affect the image obtained; they do not prove that it depicts a uniquely identified concept. For example, maximize_channel(extractor, 21, 0) targets channel 0 at index 21, provided the extractor includes that layer. Save the returned [0, 1] RGB tensor with torchvision.utils.save_image. Try multiple initial images and compare with highly activating patches from real data.

def total_variation(x):
    vertical = (x[:, :, 1:] - x[:, :, :-1]).abs().mean()
    horizontal = (x[:, :, :, 1:] - x[:, :, :, :-1]).abs().mean()
    return vertical + horizontal

def maximize_channel(extractor, layer, channel, size=64, steps=100):
    """Optimize one image; the extractor must be frozen and in evaluation mode."""
    if not isinstance(size, int) or size < 16 or not isinstance(steps, int) or steps < 0:
        raise ValueError("Use size at least 16 and nonnegative integer steps")
    ref = extractor.mean
    image = torch.rand(1, 3, size, size, device=ref.device, dtype=ref.dtype,
                       requires_grad=True)
    optimizer = torch.optim.Adam([image], lr=0.03)
    for _ in range(steps):
        optimizer.zero_grad(set_to_none=True)
        activation = extractor(image)[layer][:, channel].mean()
        loss = -activation + 0.01 * total_variation(image) + 0.001 * image.square().mean()
        loss.backward()
        optimizer.step()
        with torch.no_grad():
            image.clamp_(0, 1)
    return image.detach()

Convolution, padding, stride, and pooling are introduced in CNN Fundamentals: From Convolution to Image Classification.

Content loss

Let \(F^\ell\) be the generated image’s activation at layer \(\ell\), reshaped into \(C\) channels by \(S=H W\) spatial positions, and let \(P^\ell\) be the content reference’s activation. Matching corresponding entries encourages similar responses at corresponding locations. Semantic similarity alone does not guarantee small feature distance. The code uses mean squared error:

\[\mathcal{L}_{\text{content}}=\frac{1}{CS}\sum_{i=1}^{C}\sum_{k=1}^{S}(F^\ell_{ik}-P^\ell_{ik})^2.\]

For feature vectors [1, 3] and [2, 1], the mean squared error is (1 + 4) / 2 = 2.5. Earlier layers tend to constrain local detail more directly; deeper layers compare responses over larger receptive fields. Neither choice guarantees exact pixels or preserved object layout. We use conv4_2, as in the original work, and tune its weight alongside the style losses. Our mean reduction differs from a half-sum convention, so numerical loss weights are not interchangeable between implementations.

The Gram matrix

A Gram matrix aggregates products of feature channels over spatial positions. The unnormalized definition is:

\[G_{ij}^{\ell}=\sum_{k}F_{ik}^{\ell}F_{jk}^{\ell}\]

Entry \((i,j)\) sums the products of channels \(i\) and \(j\) at the same locations. This measures uncentered second-order statistics, not Pearson correlation: means are not subtracted, and there is no standard-deviation normalization. Channel names such as “blue” or “swirl” are not generally known, so a large entry does not prove the presence of a named visual motif. A common permutation of all channels’ spatial positions leaves the matrix unchanged, although individual CNN features can encode patterns within their receptive fields.

def gram_matrix(feat):
    """(N, C, H, W) -> (N, C, C), divided by C*H*W."""
    n, c, h, w = feat.shape
    f = feat.reshape(n, c, h * w)
    return (f @ f.transpose(1, 2)) / (c * h * w)

def style_loss(gen_feats, style_grams, weights):
    terms = [weights[layer] * F.mse_loss(gram_matrix(gen_feats[layer]), target)
             for layer, target in style_grams.items()]
    return torch.stack(terms).sum()

example = torch.tensor([[[[1., 2.]], [[3., 4.]]]])
print("normalized Gram", gram_matrix(example)[0].tolist())
# normalized Gram [[1.25, 2.75], [2.75, 6.25]]

The two channels above are [1, 2] and [3, 4]. Their product sums are 5, 11, and 25; dividing by \(CS=4\) gives [[1.25, 2.75], [2.75, 6.25]]. This normalization removes growth from repeating the same spatial statistics at more positions and introduces a channel-count scale. It does not equalize activation magnitudes across layers. We also average squared differences over the \(C^2\) Gram entries; together these choices scale a layer’s unnormalized squared Gram difference by \(1/(C^4S^2)\) when the image sizes match. Layer weights must be chosen for this convention.

Matching several layers compares feature products at several receptive-field scales. Shallow layers emphasize relatively local patterns; deeper layers can respond to larger motifs. The loss cannot uniquely recover a style image or guarantee particular colors or composition. The content term retains a spatial constraint, so losing explicit position in the style descriptor does not remove all spatial information from the complete objective.

Optimizing the image

\[\mathcal{L}=\alpha\mathcal{L}_{\text{content}}+\beta\mathcal{L}_{\text{style}}\]

def style_transfer(content, style, extractor, content_layer=21,
                   style_layers=(0, 5, 10, 19, 28),
                   alpha=1.0, beta=1e6, steps=300, lr=0.03):
    """Single RGB images in [0, 1]; return raw RGB, not normalized pixels."""
    style_layers = tuple(style_layers)
    if not style_layers or not set(style_layers + (content_layer,)) <= extractor.layers:
        raise ValueError("Extractor must include every requested loss layer")
    if not isinstance(steps, int) or steps < 0 or not math.isfinite(lr) or lr <= 0:
        raise ValueError("Use nonnegative integer steps and a positive finite learning rate")
    if not all(math.isfinite(v) and v >= 0 for v in (alpha, beta)) or alpha + beta == 0:
        raise ValueError("Loss weights must be finite, nonnegative, and not both zero")
    for image in (content, style):
        if image.ndim != 4 or image.shape[:2] != (1, 3) or min(image.shape[2:]) < 16:
            raise ValueError("Expected one RGB image with height and width at least 16")
        if not torch.isfinite(image).all() or image.min() < 0 or image.max() > 1:
            raise ValueError("Images must contain finite values in [0, 1]")
        if image.device != extractor.mean.device or image.dtype != extractor.mean.dtype:
            raise ValueError("Images and extractor must share device and dtype")
    extractor.eval().requires_grad_(False)
    generated = content.detach().clone().requires_grad_(True)
    optimizer = torch.optim.Adam([generated], lr=lr)
    with torch.no_grad():
        content_target = extractor(content)[content_layer]
        style_feats = extractor(style)
        style_grams = {layer: gram_matrix(style_feats[layer]) for layer in style_layers}
    weights = {layer: 1.0 / len(style_layers) for layer in style_layers}
    for _ in range(steps):
        optimizer.zero_grad(set_to_none=True)
        feats = extractor(generated)
        c = F.mse_loss(feats[content_layer], content_target)
        s = style_loss(feats, style_grams, weights)
        loss = alpha * c + beta * s
        if not torch.isfinite(loss):
            raise RuntimeError("Nonfinite objective; inspect inputs and loss scales")
        loss.backward()
        optimizer.step()
        with torch.no_grad():
            generated.clamp_(0, 1)
    return generated.detach()

The optimizer contains only the generated image. Freezing extractor parameters still allows gradients to pass through its operations to that image; wrapping the generated-image forward pass in no_grad() would break the computation. Target features can be computed without gradients. We start from a detached copy of the content image, then project pixels back into [0, 1] after each Adam update. Here steps counts optimizer updates exactly. This bounded optimization is not guaranteed to decrease the loss on every update or find a global minimum.

L-BFGS is also commonly used, but one optimizer call may invoke its closure several times; counting closure calls as image updates gives a misleading iteration budget. Neither optimizer is universally faster. The relative loss weights, layer choices, normalization, image resolution, initialization, learning rate, and optimization budget all affect the result. The defaults above are starting values, not a calibrated scale from weak to strong stylization.

The file wrapper below preserves each image’s aspect ratio and limits its longest side. Content and style can have different spatial dimensions because their Gram matrices have the same channel dimensions. The output keeps the resized content dimensions. Normalization happens inside FeatureExtractor, so saving needs no inverse normalization. Call stylize_files("content.jpg", "style.jpg", "stylized.png") with your own images; the first call may download VGG-19 weights.

def stylize_files(content_path, style_path, output_path, max_side=256, steps=300):
    from PIL import Image, ImageOps
    from torchvision.transforms.functional import pil_to_tensor
    from torchvision.utils import save_image
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    def load(path):
        with Image.open(path) as source:
            image = ImageOps.exif_transpose(source).convert("RGB")
            image.thumbnail((max_side, max_side), Image.Resampling.LANCZOS)
            return (pil_to_tensor(image).float() / 255).unsqueeze(0).to(device)
    content, style = load(content_path), load(style_path)
    extractor = FeatureExtractor((0, 5, 10, 19, 21, 28)).to(device).eval()
    result = style_transfer(content, style, extractor, steps=steps)
    save_image(result.cpu(), output_path)
    return result

For an offline check of the optimization mechanics, use the small random feature network below. Its loss can decrease and its pixels can change without producing a meaningful artistic style. This run checks the code path; visual quality requires the pretrained extractor and actual reference images.

torch.manual_seed(3)
toy_features = nn.Sequential(nn.Conv2d(3, 4, 3, padding=1), nn.ReLU(inplace=True),
                             nn.Conv2d(4, 4, 3, padding=1))
toy = FeatureExtractor((0, 2), features=toy_features).eval()
content_demo = torch.rand(1, 3, 16, 16)
style_demo = torch.rand(1, 3, 20, 18) * 0.5
with torch.no_grad():
    target_demo = toy(content_demo)[2]
    grams_demo = {0: gram_matrix(toy(style_demo)[0])}
def demo_objective(image):
    feats = toy(image)
    return F.mse_loss(feats[2], target_demo) + 100 * style_loss(feats, grams_demo, {0: 1.0})
with torch.no_grad():
    before = demo_objective(content_demo).item()
result_demo = style_transfer(content_demo, style_demo, toy, content_layer=2,
                             style_layers=(0,), beta=100, steps=40, lr=0.02)
with torch.no_grad():
    after = demo_objective(result_demo).item()
print(f"toy objective {before:.6f} -> {after:.6f}")
print("mean pixel change", round((result_demo - content_demo).abs().mean().item(), 6))
print("extractor gradients", any(p.grad is not None for p in toy.parameters()))
# toy objective 1.425026 -> 0.061215
# mean pixel change 0.226164
# extractor gradients False

Making it fast

Direct pixel optimization requires repeated forward and backward passes for each output image. Its runtime depends on resolution, hardware, and the chosen budget. Feed-forward methods move much of that optimization cost into training a reusable image transformation network.

Fast style transfer trains a network whose output is evaluated with a frozen perceptual extractor. The transform network’s weights receive gradients through the extractor. A fixed-style network can then produce an output in one forward pass. Early systems trained one network per style; conditional and arbitrary-style systems use other ways to supply style information.

Adaptive instance normalization (AdaIN) adjusts each content feature channel using the style feature channel’s mean and standard deviation over spatial positions: \[\operatorname{AdaIN}(x,y)=\sigma(y)\frac{x-\mu(x)}{\sigma(x)}+\mu(y).\] This formula assumes nonzero content standard deviation; implementations stabilize the denominator. It matches per-channel means and variances, not cross-channel covariances, so it is not full Gram matching or a first-order approximation to it. A decoder must be trained to map the adjusted features back to images. Once trained, that decoder can accept new reference styles without per-style retraining; the AdaIN operation alone cannot generate an RGB image.

Matching channel statistics is useful but incomplete. For example, channels [−1, 1] and [−1, 1] have the same means and variances as [−1, 1] and [1, −1]. Their cross-channel product averages are +1 and −1. AdaIN cannot change the sign of the first pair’s correlation by independent positive rescaling. Stabilization also means exact variance matching can fail for a constant or nearly constant content channel.

Related normalization operations are compared in Normalization Layers Explained: BatchNorm, LayerNorm, GroupNorm, RMSNorm.

Where feature-based losses are useful

Reference-image style transfer and text-conditioned generation offer different controls. A content/style pair specifies a spatial reference and a visual reference; a text prompt supplies another form of conditioning. Generative models can also use reference images. These methods coexist, and choosing among them depends on how much structure must be preserved and what inputs are available.

Perceptual losses compare representations of images and are used in tasks such as super-resolution and image translation. Their usefulness depends on the features and distance chosen. Gram matrices provide one compact texture descriptor. Optimizing an input through a fixed network also appears in feature visualization and adversarial examples; what differs is the objective and the constraints on the input.

Denoising-based image generation is introduced in Diffusion Models Explained: DDPM, Samplers, and Guidance.

Methods for inspecting what a model learned are covered in Interpretability: Saliency, Grad-CAM, Probing, and Circuits.

Exercises

1. The Gram matrix discards position. Compute the Gram matrix of a feature map and of the same feature map with its spatial positions shuffled. Report the difference and explain what this means for style.

You should get: two Gram matrices that are identical to floating-point precision.

torch.manual_seed(0)
feat = torch.randn(1, 16, 8, 8)
perm = torch.randperm(64)
shuffled = feat.flatten(2)[:, :, perm].reshape_as(feat)
diff = (gram_matrix(feat) - gram_matrix(shuffled)).abs().max()
print(f"max difference {diff:.2e}")
# max difference 1.49e-08

2. The optimization target is the image. Explain what has requires_grad=True in classical style transfer, what the optimizer updates, and why the VGG weights are frozen. Then say what changes in a feed-forward style transfer network.

You should get: an optimizer whose parameter list contains a single tensor that is not a weight.

import torch
img = torch.randn(1, 3, 224, 224, requires_grad=True)   # the variable
opt = torch.optim.LBFGS([img])                          # optimizes the image
print(sum(p.numel() for p in [img]))
# 150528

3. Comparing layer choices. Explain what changes when the content loss is taken from an early convolutional layer versus a deep one, and why the style loss is summed across several layers instead of taken from one.

You should get: a distinction between local and broader feature constraints, with no guaranteed visual outcome.

References

• Zeiler and Fergus (2014). Visualizing and Understanding Convolutional Networks. ECCV.
• Gatys, Ecker, and Bethge (2016). Image Style Transfer Using Convolutional Neural Networks. CVPR.
• Johnson, Alahi, and Fei-Fei (2016). Perceptual Losses for Real-Time Style Transfer and Super-Resolution. ECCV.
• Ulyanov, Vedaldi, and Lempitsky (2016). Instance Normalization: The Missing Ingredient for Fast Stylization. arXiv preprint.
• Huang and Belongie (2017). Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization. ICCV.
• Olah, Mordvintsev, and Schubert (2017). Feature Visualization. Distill.