Classifier-Free Guidance

Classifier-free guidance (CFG) is a sampling-time technique for conditional diffusion models. At each denoising step the model is run twice, once with the condition (for example a text prompt) and once without it, and the two predictions are extrapolated: the sample is pushed further in the direction the condition adds. A guidance scale controls how far. Moderately higher scales can make images follow the prompt more closely and look more “typical”, at the cost of diversity; scales that are too high distort the samples and lower image quality.

The update

A diffusion denoiser predicts the noise \(\varepsilon\) in a noisy input \(x_t\) at step \(t\). Write \(\varepsilon_\theta(x_t,c)\) for the prediction given the condition \(c\) and \(\varepsilon_\theta(x_t,\varnothing)\) for the prediction with the condition dropped. CFG, introduced by Ho and Salimans, samples with

\[ \tilde\varepsilon = \varepsilon_\theta(x_t,\varnothing) + w\,\big(\varepsilon_\theta(x_t,c)-\varepsilon_\theta(x_t,\varnothing)\big). \]

With \(w=0\) this is the unconditional model, with \(w=1\) the ordinary conditional prediction, and with \(w>1\) the conditional prediction is extrapolated past itself. (Ho and Salimans write the same rule as \((1+w’)\varepsilon_c – w’\varepsilon_\varnothing\), so \(w’ = w-1\); libraries differ, and “guidance scale 7.5” normally means \(w=7.5\) in the form above.) A scalar example with \(\varepsilon_\varnothing=0.20\) and \(\varepsilon_c=0.50\) gives 0.20 at \(w=0\), 0.50 at \(w=1\), 1.10 at \(w=3\) and 2.45 at \(w=7.5\).

One network supplies both predictions. During training the condition is replaced by a null token for a fraction of examples (condition dropout), so the same weights learn the conditional and the unconditional task, and no separate classifier is needed. This is what “classifier-free” refers to: earlier classifier guidance trained an image classifier on noisy images and added its gradient.

Why the extrapolation works

The difference \(\varepsilon_c-\varepsilon_\varnothing\) is, up to a negative scale factor, the gradient of the log-probability that the condition is present given the image, \(\nabla_x\log p(c\mid x)\). It is the direction that makes \(x\) look more as if it were made for \(c\). Scaling it by \(w\) corresponds to sampling from a distribution proportional to \(p(x)\,p(c\mid x)^w\). With \(w=1\) that is the true conditional \(p(x\mid c)\); with \(w>1\) the sharpened term \(p(c\mid x)^w\) up-weights images the implicit classifier is very sure about. (Applying the rule at every noise level is an approximation to that product distribution, not an exact sampler of it.)

Fidelity against diversity

The exact one-dimensional case shows the trade without a trained network. The unconditional data is an even mixture of \(N(-1,1)\) and \(N(1,1)\), the condition selects the right-hand component \(N(1,1)\), and the guided density \(\propto p(x)^{1-w}p(x\mid c)^w\) is computed on a grid:

import numpy as np
x = np.linspace(-6, 6, 24001)
N = lambda m, s: np.exp(-0.5 * ((x - m) / s) ** 2) / (s * np.sqrt(2 * np.pi))
pc = N(1, 1)                              # p(x | c): the right-hand component
pu = 0.5 * N(-1, 1) + 0.5 * N(1, 1)       # p(x): unconditional mixture
print("w   P(x>0)  mean   std  mean log p_c(x) of samples")
for w in (0, 1, 2, 4, 8):
    lg = (1 - w) * np.log(pu + 1e-300) + w * np.log(pc + 1e-300)    # guided density, up to a constant
    p = np.exp(lg - lg.max()); p /= p.sum()
    m = (p * x).sum(); s = np.sqrt((p * (x - m) ** 2).sum())
    print(f"{w:<3} {p[x > 0].sum():6.3f} {m:6.3f} {s:6.3f}  {(p * np.log(pc + 1e-300)).sum():8.3f}")

w   P(x>0)  mean   std  mean log p_c(x) of samples
0    0.500 -0.000  1.414    -2.419
1    0.841  1.000  1.000    -1.419
2    0.940  1.290  0.863    -1.333
4    0.989  1.539  0.767    -1.359
8    1.000  1.770  0.699    -1.460

At \(w=1\) the samples match the conditional exactly (mean 1, standard deviation 1). Raising \(w\) moves more of them onto the right-hand side (0.841 up to 1.000 at \(w=8\)) and narrows the spread (1.0 down to 0.70), but the mean moves past the true conditional mean of 1 to 1.77. The last column is the average log-density of the guided samples under the true conditional: it improves up to \(w=2\) and then declines, because the samples are being pushed into the outer tail beyond the data. Images in a real model show the corresponding effects: stronger adherence to the prompt, lower variety, and over-saturated or exaggerated features at high scales. Real conditional models are imperfect, so a scale above 1 can repair weak prompt adherence, which is why practical scales are well above 1.

Cost and negative prompts

CFG needs two network evaluations per step, usually batched together, which roughly doubles sampling compute. Some distilled models bake the guidance into a single pass.

A negative prompt replaces the unconditional branch by the prediction for an unwanted condition \(c_-\):

\[ \tilde\varepsilon = \varepsilon_\theta(x_t,c_-) + w\,\big(\varepsilon_\theta(x_t,c)-\varepsilon_\theta(x_t,c_-)\big). \]

The direction now points away from \(c_-\) and toward \(c\). With \(\varepsilon_c=0.50\), \(\varepsilon_\varnothing=0.20\), a negative-prompt prediction of 0.35 and \(w=3\), plain CFG gives 1.10 and the negative-prompt form 0.80, so the negative prompt makes a smaller step than the unconditional baseline in this example. This is how many text-to-image tools implement negative prompts; a given product may differ.

Neighbors

CFG is not a training change to the model’s objective beyond condition dropout, and it is not the same as temperature in language models. Temperature reshapes one distribution; guidance combines two. Guidance is also not a safety filter: a negative prompt biases sampling away from a concept but does not guarantee its absence.

References

• Ho and Salimans, Classifier-Free Diffusion Guidance — https://arxiv.org/abs/2207.12598