Neural Style Transfer: Cost Function and Image Optimization

ByJu Yeon Eum

Neural style transfer generates an image by solving an optimization problem.

Given:

  • A content image \(C\)
  • A style image \(S\)
  • A generated image \(G\)

the algorithm defines a cost function that measures how successfully \(G\) combines the content of \(C\) with the style of \(S\).

The generated image is then found by minimizing this cost with respect to its pixel values.

Problem Formulation

The objective is:\[ (C,S) \longrightarrow G \]

The generated image should:

  1. Preserve the semantic structure of \(C\)
  2. Reproduce visual statistics associated with \(S\)

These two goals are encoded in separate loss terms.

Total Cost Function

The total neural style transfer objective is:\[ J(G) = \alpha J_{\text{content}}(C,G) + \beta J_{\text{style}}(S,G) \]

where:

  • \(J_{\text{content}}(C,G)\) measures content similarity.
  • \(J_{\text{style}}(S,G)\) measures style similarity.
  • \(\alpha\) controls content strength.
  • \(\beta\) controls style strength.

An optional image-regularization term can also be included:\[ J(G) = \alpha J_{\text{content}}(C,G) + \beta J_{\text{style}}(S,G) + \gamma J_{\text{TV}}(G) \]

where \(J_{\text{TV}}\) encourages spatial smoothness.

Why Use Two Weights?

If the objective contains only content and style terms, their relative importance is primarily controlled by:\[ \frac{\beta}{\alpha} \]

Therefore, one relative-weight parameter could theoretically be sufficient.

Using two weights is still convenient because it makes the formulation symmetric and allows the loss scales to be adjusted explicitly.

The absolute values can also matter in practice because they interact with:

  • Learning rate
  • Optimizer behavior
  • Gradient magnitude
  • Additional regularization terms
  • Numerical precision

A Fixed Pretrained ConvNet

The content and style losses are not normally calculated from raw pixels alone. Instead, a pretrained convolutional network extracts visual features.

Let:\[ \phi_\theta^{[\ell]}(X) \]

represent the activation at layer \(\ell\) when image \(X\) passes through the network.

During classical neural style transfer:\[ \theta=\text{fixed} \]

The pretrained model is not updated.

Only the generated image is optimized:\[ G=\text{trainable variable} \]

Neural style transfer uses backpropagation through the network, but it updates the image pixels rather than the network parameters.

Content Cost

Choose a content layer \(\ell_c\). Compute:\[ A_C^{[\ell_c]} = \phi^{[\ell_c]}(C) \]

and:\[ A_G^{[\ell_c]} = \phi^{[\ell_c]}(G) \]

The content loss can be defined as:\[ J_{\text{content}}(C,G) = \frac{1}{2} \left\| A_G^{[\ell_c]} – A_C^{[\ell_c]} \right\|_F^2 \]

Minimizing this term encourages \(G\) to produce a deep representation similar to that of the content image.

A shallow content layer preserves fine local details, while a deeper layer permits greater visual transformation.

Style Cost

Style is commonly represented by feature-channel co-activations.

For style layer \(\ell\), reshape its activation tensor into:\[ F^{[\ell]}(X) \in \mathbb{R}^{C_\ell\times H_\ell W_\ell} \]

The Gram matrix is:\[ \Gamma^{[\ell]}(X) = F^{[\ell]}(X)F^{[\ell]}(X)^T \]

It summarizes how strongly different feature channels activate together.

A single-layer style loss is:\[ J_{\text{style}}^{[\ell]}(S,G) = \frac{ \left\| \Gamma^{[\ell]}(G) – \Gamma^{[\ell]}(S) \right\|_F^2 }{ 4C_\ell^2(H_\ell W_\ell)^2 } \]

Using several layers:\[ J_{\text{style}}(S,G) = \sum_{\ell\in\mathcal{L}_S} \lambda_\ell J_{\text{style}}^{[\ell]}(S,G) \]

Early layers capture fine colors and textures, while deeper layers capture larger and more abstract visual patterns.

Initializing the Generated Image

Before optimization, \(G\) must be initialized.

Random-Noise Initialization

Initialize every pixel randomly:\[ G_0\sim\mathcal{D}_{\text{noise}} \]

The initial image resembles visual noise. As optimization proceeds, structure and style emerge.

This initialization gives the algorithm substantial freedom but may require more iterations.

Content Initialization

Set:\[ G_0=C \]

This often:

  • Preserves content more strongly
  • Converges faster
  • Produces stable structure
  • Requires fewer optimization steps

Mixed Initialization

A compromise is:\[ G_0 = \rho C + (1-\rho)\epsilon \]

where:

  • \(\epsilon\) is random noise.
  • \(0\leq\rho\leq1\)

Gradient-Based Image Optimization

At optimization step \(t\), the generated image is:\[ G_t \]

Calculate:\[ J(G_t) \]

and its pixel gradient:\[ \nabla_GJ(G_t) = \frac{\partial J(G_t)}{\partial G_t} \]

A gradient-descent update is:\[ G_{t+1} = G_t – \eta \nabla_GJ(G_t) \]

where \(\eta\) is the learning rate.

Every pixel and color channel can be adjusted:\[ G \in \mathbb{R}^{H\times W\times3} \]

For an image of size \(500\times500\), this means optimizing:\[ 500\times500\times3 = 750{,}000 \]

pixel variables.

What Backpropagation Does

The generated image is passed through the fixed feature extractor:\[ G \rightarrow \phi^{[\ell]}(G) \rightarrow J(G) \]

Backpropagation calculates:\[ \frac{\partial J}{\partial G} \]

through all selected network layers.

The gradient indicates how every pixel should change to:

  • Reduce content-feature differences
  • Reduce style-statistic differences
  • Improve spatial smoothness when regularization is used

Optimization Procedure

The complete process is:

load a fixed pretrained convolutional network
    ↓
preprocess the content and style images
    ↓
compute and store content target features
    ↓
compute and store style target Gram matrices
    ↓
initialize the generated image
    ↓
compute generated-image features
    ↓
calculate content and style losses
    ↓
backpropagate to the generated pixels
    ↓
update the generated image
    ↓
repeat until satisfactory

Precomputing Fixed Targets

Because \(C\) and \(S\) remain unchanged, their target features should be calculated once.

Precompute:\[ A_C^{[\ell_c]} \]

and:\[ \Gamma_S^{[\ell]} \quad \text{for every selected style layer} \]

These tensors should be treated as constants.

At each optimization step, only features derived from \(G\) must be recomputed.

Optional Total-Variation Loss

Pixel optimization can introduce noisy local changes. Total-variation regularization penalizes excessive variation between neighboring pixels.

One form is:\[ J_{\text{TV}}(G) = \sum_{i,j} \left|G_{i+1,j}-G_{i,j}\right| + \left|G_{i,j+1}-G_{i,j}\right| \]

Another uses squared differences:\[ J_{\text{TV}}(G) = \sum_{i,j} \left( G_{i+1,j}-G_{i,j} \right)^2 + \left( G_{i,j+1}-G_{i,j} \right)^2 \]

This can reduce high-frequency artifacts and produce smoother images.

Choosing the Content–Style Balance

Strong Content Preservation

Increase the relative content influence:\[ \frac{\alpha}{\beta} \]

The result tends to retain:

  • Original layout
  • Object shapes
  • Fine scene structure

The style effect may become subtle.

Strong Stylization

Increase:\[ \frac{\beta}{\alpha} \]

The result tends to show:

  • Stronger texture
  • More dramatic color changes
  • Larger stylistic patterns
  • Greater structural distortion

The best balance is subjective and depends on the selected layers and normalization.

Optimization Algorithms

Plain gradient descent can minimize the objective, but practical implementations often use:

  • Adam
  • L-BFGS

Adam is straightforward and provides flexible step-by-step control.

L-BFGS can converge effectively for deterministic image-optimization problems but may be more sensitive to implementation details and memory use.

Because the optimization variable is one image rather than millions of model parameters, either method can work well.

Image Constraints

After each update, the generated tensor may leave the valid range expected by the model or image format.

Possible strategies include:

Pixel Clamping

Constrain values after every step:\[ G \leftarrow \operatorname{clip}(G,G_{\min},G_{\max}) \]

Reparameterization

Optimize an unconstrained tensor \(Z\) and transform it:\[ G = G_{\min} + (G_{\max}-G_{\min})\sigma(Z) \]

This keeps pixel values within bounds automatically.

Clamping is simpler, while reparameterization provides a smooth bounded transformation.

Preprocessing Consistency

The content, style, and generated images must use identical preprocessing.

This may include:

  • RGB or BGR channel order
  • Pixel scaling
  • Mean subtraction
  • Standard-deviation normalization
  • Image resizing
  • Batch and channel dimensions

If the feature extractor expects normalized inputs, the generated image must be represented in that same normalized space during feature calculation.

The final tensor must then be converted back into a displayable image.

Simplified PyTorch-Style Optimization

import torch


generated = content_image.clone().detach()
generated.requires_grad_(True)

optimizer = torch.optim.Adam(
    [generated],
    lr=0.02
)

for step in range(num_steps):
    optimizer.zero_grad()

    generated_activations = feature_extractor(generated)

    content = compute_content_loss(
        generated_activations,
        target_content
    )

    style = compute_style_loss(
        generated_activations,
        target_style_grams
    )

    total_variation = compute_total_variation(generated)

    total = (
        content_weight * content
        + style_weight * style
        + variation_weight * total_variation
    )

    total.backward()
    optimizer.step()

    with torch.no_grad():
        generated.clamp_(minimum_value, maximum_value)

The feature extractor’s parameters should have gradient updates disabled. Only generated is passed to the optimizer.

Why the Image Improves Gradually

A randomly initialized image initially has:

  • Incorrect content features
  • Incorrect style statistics
  • Little coherent structure

As optimization proceeds:

  1. Broad content structure begins to appear.
  2. Large color relationships shift toward the style image.
  3. Texture patterns develop.
  4. Fine details become more coherent.
  5. The total objective decreases.

The process may not decrease every individual loss monotonically because content and style objectives compete.

History of the Method

The optimization-based formulation of neural style transfer was introduced by Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge.

Its major conceptual contribution was demonstrating that a pretrained convolutional network could separate:

  • Content through deeper feature activations
  • Style through feature-channel statistics

The result is generated by optimizing an image in feature space rather than applying a predefined artistic filter.

Common Mistakes

Updating the Feature Extractor

The network should remain fixed. The generated image is the optimized variable.

Forgetting to Enable Gradients for \(G\)

The generated tensor must participate in automatic differentiation.

Tracking Gradients Through Target Features

Content targets and style Gram matrices should be detached because they remain constant.

Using Inconsistent Image Preprocessing

All three images must be processed according to the same model convention.

Assuming Loss Weights Transfer Across Implementations

Different Gram-matrix and content-loss normalizations change the useful values of \(\alpha\) and \(\beta\).

Using an Excessive Learning Rate

Large updates can introduce unstable colors, severe artifacts, or oscillation.

Failing to Constrain Pixel Values

Unbounded optimization may produce extreme values that are invalid for display or inappropriate for the feature extractor.

Key Takeaway

Neural style transfer defines the generated image as the solution to an optimization problem:\[ G^* = \underset{G}{\operatorname{argmin}} \left[ \alpha J_{\text{content}}(C,G) + \beta J_{\text{style}}(S,G) \right] \]

A fixed pretrained convolutional network evaluates the content and style of \(G\). Backpropagation computes gradients with respect to the image itself:\[ G \leftarrow G-\eta\frac{\partial J}{\partial G} \]

By repeatedly updating its pixels, the generated image develops content features similar to \(C\) and style statistics similar to \(S\).

Similar Posts

Questions, corrections, or additional insights?