Image and Video Generation

Ask an image generator for a red cube on top of a blue sphere and four things can go wrong that an overall sample-quality number will not show. The picture is sharp but the colors are swapped. The cube sits beside the sphere instead of on it. One seed honors the prompt and the next ignores it. Or the image is fine, and nobody downstream can tell whether it came from a camera or a model. Video adds a fifth failure: every frame is plausible and the clip still flickers.

Each failure maps to an engineering decision. Whether the colors stay attached to the right objects depends on how the prompt reaches the network. Whether the clip is stable in time depends on how frames are allowed to see each other. Whether a new model is better than the old one depends on metrics, prompt sets and statistics. Whether the output can be traced or constrained depends on provenance marks and safety layers around the model. The sampler, noise schedule and basic classifier-free guidance are covered in Diffusion Models Explained: DDPM, Samplers, and Guidance; everything below builds on a denoiser that takes a noisy input, a timestep and a condition, and the questions are about the condition, the stack of frames, the measurement and the controls.

What a conditioned denoiser receives

A conditional diffusion model learns a noise predictor \(\varepsilon_\theta(x_t, t, c)\). The input \(x_t\) is the noisy image or latent, \(t\) says how noisy it is, and \(c\) is everything else the user or system supplies: text, a reference image, a depth map, a mask, a camera path, an audio track. The network can only use \(c\) through whatever pathway the architecture provides, so the pathway determines what kind of control is possible.

Four pathways cover most systems.

Pathway	How c enters	Typical conditions	What it controls well	Where it struggles
Channel concatenation	c is resized to the latent grid and stacked with x_t as extra input channels	Mask plus masked image (inpainting), low-resolution image, depth or edge map	Pixel-aligned structure; the network sees c at every location	Needs paired training data; a new condition type means retraining or a new input layer
Feature modulation	A vector computed from c scales and shifts normalized activations (FiLM or adaptive normalization)	Timestep, class label, pooled text or image embedding, camera parameters	Global attributes: style, class, overall layout statistics	A single vector cannot say which attribute belongs to which object
Cross-attention	Latent positions form queries; tokens of c supply keys and values	Token sequences: text embeddings, image-patch embeddings	Different positions can read different tokens, so prompts with several objects are possible	Cost grows with the number of condition tokens; binding of attributes to objects is learned, not guaranteed
Side branch or adapter	A trainable network reads the condition and adds features into a frozen base model	Edges, pose, depth, segmentation, reference images	Adds a new control to an existing model without retraining it	Quality of the control depends on the branch’s training data; several controls interact

Two further mechanisms act outside the network. Guidance changes how predictions are combined at sampling time. Starting the sampler from a noised version of an existing image, instead of pure noise, makes the source image itself a condition. Both are covered below because they are the controls users reach for first.

Text conditioning with cross-attention

A text prompt is turned into a sequence of embedding vectors by a text encoder; how such encoders are trained and aligned with images is the subject of Multimodal Models: Images, Audio, and Cross-Modal Alignment, and CLIP-style contrastive training is introduced in Self-Supervised Learning: SimCLR, BYOL, MAE, and CLIP. For the denoiser, the prompt is a matrix of token vectors. Cross-attention lets each spatial position of the latent consult that matrix.

The mechanism is the attention operation from the Transformer, with the roles split between two sequences. Each latent position \(i\) produces a query \(q_i\). Each text token \(j\) produces a key \(k_j\) and a value \(v_j\). The position’s update is \(\sum_j a_{ij} v_j\) with \(a_{ij}=\mathrm{softmax}_j(q_i^\top k_j/\sqrt{d})\). The softmax runs over tokens, separately for every position, so every position receives a convex combination of token values, and the weights decide which words matter at that place in the image. Self-attention over the latent, which lets positions exchange information with each other, sits in the same blocks; if the Transformer machinery is unfamiliar, Transformers from Scratch: Self-Attention to Encoder–Decoder builds it step by step.

The following listing builds the smallest instance of this idea. The latent is a 6×6 grid, the prompt has five tokens, and the queries are written by hand so that the left half of the grid asks for the tokens “red” and “cube” and the right half asks for “blue” and “sphere”. Real models learn queries, keys and values; the hand-built version isolates the shape of the computation and the way attention mass splits across the grid.

import numpy as np

rng = np.random.default_rng(0)
d = 16
tokens = ["<s>", "red", "cube", "blue", "sphere"]
K = rng.normal(size=(5, d)) / np.sqrt(d) * 2.0           # one key per text token
V = rng.normal(size=(5, d))                              # one value per text token
Q = np.zeros((6, 6, d))                                  # one query per latent position (6x6 grid)
for i in range(6):
    for j in range(6):
        target = (1, 2) if j < 3 else (3, 4)             # hand-built: left half asks for "red cube", right half for "blue sphere"
        Q[i, j] = (K[target[0]] + K[target[1]]) * 2.0 + rng.normal(size=d) * 0.3

logits = Q.reshape(-1, d) @ K.T / np.sqrt(d)             # (36 positions, 5 tokens)
A = np.exp(logits - logits.max(1, keepdims=True))
A /= A.sum(1, keepdims=True)                             # softmax over tokens, separately for every position
update = A @ V                                           # text-derived update added at every position

print("attention shape", A.shape, "| each row sums to", np.round(A.sum(1)[:3], 3))
print("update shape", update.shape)
for n, tok in enumerate(tokens):
    m = A[:, n].reshape(6, 6)
    print(f"{tok:>6}: attention mass on left half {m[:, :3].sum():5.2f}   right half {m[:, 3:].sum():5.2f}")

Program output (NumPy 2.5.3, Python 3.12; random vectors with a fixed seed):

attention shape (36, 5) | each row sums to [1. 1. 1.]
update shape (36, 16)
   <s>: attention mass on left half  1.88   right half  0.84
   red: attention mass on left half  2.89   right half  1.12
  cube: attention mass on left half  8.35   right half  1.93
  blue: attention mass on left half  1.85   right half  4.86
sphere: attention mass on left half  3.03   right half  9.25

The attention matrix has one row per latent position and one column per token, and each row sums to one. The update has one 16-dimensional vector per position, which the network adds to its features, so text influence is spatially varying by construction. The mass columns show the effect of the queries: “cube” collects 8.35 of the 18 units of attention on the left half and “sphere” collects 9.25 of 18 on the right. The split is not clean, because attention is a soft weighting driven by dot products with random keys, and “red” receives less mass on the left than “cube” does. Meaning is not stored in the weights; only the match between the position’s query and each token’s key is. Which words end up controlling which regions is a learned property of the whole model, and the failure where colors swap is a failure of exactly this learned assignment.

Two practical consequences follow. First, the prompt length that the text encoder accepts and the number of tokens attended to bound how much detail a prompt can carry. Second, because the softmax normalizes over tokens, a rare or poorly encoded word competes with common words for the same unit of attention at each position; a prompt with many objects divides that unit among them.

Guidance with several conditions and negative prompts

Classifier-free guidance combines an unconditional prediction \(\varepsilon_\theta(x_t,\varnothing)\) and a conditional one \(\varepsilon_\theta(x_t,c)\), with a scale \(w\) that extrapolates between them; the formula and the training trick are in Diffusion Models Explained: DDPM, Samplers, and Guidance. Production prompts rarely have one condition. A user supplies a text, a reference image, a depth map, and a negative prompt. How to combine several predictions follows from what the predictions mean.

For a model trained with Gaussian corruption \(x_t=\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\,\varepsilon\), the ideal noise predictor is related to the score of the noisy marginal by \(\varepsilon^*(x_t)=-\sqrt{1-\bar\alpha_t}\,\nabla_{x_t}\log p_t(x_t)\). This is an identity for the ideal denoiser; a trained network approximates it. Applying it to the conditional distribution and using Bayes’ rule, \(\nabla\log p_t(x_t\mid c)=\nabla\log p_t(x_t)+\nabla\log p_t(c\mid x_t)\), gives

\[\varepsilon^*(x_t,c)-\varepsilon^*(x_t,\varnothing)=-\sqrt{1-\bar\alpha_t}\;\nabla_{x_t}\log p_t(c\mid x_t)\]

The difference between the conditional and unconditional predictions is the gradient of an implicit classifier that says how much the noisy input \(x_t\) looks like it was made for \(c\). Guidance with scale \(w\) multiplies that term, which corresponds to sampling from a distribution proportional to \(p(x)\,p(c\mid x)^w\). Two consequences are useful for practice.

• Composition. If two conditions are independent given the image, \(p(c_1,c_2\mid x)=p(c_1\mid x)\,p(c_2\mid x)\), their implicit classifiers add: \(\tilde\varepsilon=\varepsilon_\varnothing+w_1(\varepsilon_{c_1}-\varepsilon_\varnothing)+w_2(\varepsilon_{c_2}-\varepsilon_\varnothing)\). InstructPix2Pix (Brooks et al.) exposes two guidance scales, one for the source image and one for the text instruction, combined in a nested form rather than the plain sum, so a user can trade faithfulness to the source against strength of the edit. The independence assumption is the weak point: "red" and "cube" are not independent given the image, because the pair describes one object.
• Negative prompts. Replacing the unconditional prediction by the prediction for an unwanted condition \(c_{-}\) gives \(\tilde\varepsilon=\varepsilon_{c_-}+w(\varepsilon_{c}-\varepsilon_{c_-})\). At \(w=1\) this equals the plain conditional prediction. For \(w>1\) it moves the sample away from \(c_-\) in the same way ordinary guidance moves it away from the unconditional model. This corresponds to a common way negative prompts are implemented in text-to-image tools; any given product may do something different.

These claims can be checked exactly in a setting where the ideal denoiser is available. The toy data has four clusters of two-dimensional points at (±2, ±2) with standard deviation 0.3. Condition “right” means the two clusters with positive x; condition “top” means the two with positive y. For a Gaussian mixture the ideal noise predictor has a closed form: the posterior probability of each cluster given the noisy point, and the posterior mean within each cluster. No network is trained, so the experiment isolates the effect of the guidance formula from the error of a learned model. The code samples with the deterministic DDIM update from the diffusion article and 50 steps, and measures precision and recall with the k-nearest-neighbor definition of Kynkäänniemi et al., which the evaluation section explains. The shared helpers come first.

import numpy as np

def ab_fn(u, s=0.008):
    f = lambda v: np.cos((v + s) / (1 + s) * np.pi / 2) ** 2
    return f(u) / f(0.0)

def knn_pr(real, gen, k=3):
    """k-NN manifold precision/recall (Kynkaanniemi et al. 2019 style)."""
    def radii(x):
        d = np.sqrt(((x[:, None] - x[None]) ** 2).sum(-1))
        return np.sort(d, 1)[:, k]
    def inside(a, b, rb):
        d = np.sqrt(((a[:, None] - b[None]) ** 2).sum(-1))
        return (d <= rb[None]).any(1).mean()
    rr, rg = radii(real), radii(gen)
    return inside(gen, real, rr), inside(real, gen, rg)   # precision, recall

import numpy as np
from common import ab_fn, knn_pr

S0 = 0.3
CENT = np.array([[2, 2], [2, -2], [-2, 2], [-2, -2]], float)   # (x sign, y sign)

def eps_hat(x, ab, comps):
    """Exact noise prediction when x0 ~ equal-weight mixture of the listed isotropic Gaussians."""
    m = CENT[comps]
    var = ab * S0**2 + (1 - ab)
    d2 = ((x[:, None, :] - np.sqrt(ab) * m[None]) ** 2).sum(-1)
    logr = -d2 / (2 * var)
    r = np.exp(logr - logr.max(1, keepdims=True)); r /= r.sum(1, keepdims=True)
    post = m[None] + (np.sqrt(ab) * S0**2 / var) * (x[:, None, :] - np.sqrt(ab) * m[None])
    x0 = (r[..., None] * post).sum(1)
    return (x - np.sqrt(ab) * x0) / np.sqrt(1 - ab)

ALL = [0, 1, 2, 3]; RIGHT = [0, 1]; TOP = [0, 2]; BOTH = [0]

def sample(eps_fn, n=4000, steps=50, seed=0):
    rng = np.random.default_rng(seed)
    x = rng.normal(size=(n, 2))
    us = np.linspace(0.999, 0.0, steps + 1)
    for i in range(steps):
        a_t, a_p = ab_fn(us[i]), ab_fn(us[i + 1])
        e = eps_fn(x, a_t)
        x0 = (x - np.sqrt(1 - a_t) * e) / np.sqrt(a_t)
        x = np.sqrt(a_p) * x0 + np.sqrt(1 - a_p) * e
    return x

def real(comps, n=2000, seed=1):
    rng = np.random.default_rng(seed)
    return CENT[rng.choice(comps, n)] + S0 * rng.normal(size=(n, 2))

def quad(x):
    q = {"right-top": ((x[:, 0] > 0) & (x[:, 1] > 0)), "right-bottom": ((x[:, 0] > 0) & (x[:, 1] < 0)),
         "left-top": ((x[:, 0] < 0) & (x[:, 1] > 0)), "left-bottom": ((x[:, 0] < 0) & (x[:, 1] < 0))}
    return {k: round(float(v.mean()), 3) for k, v in q.items()}

print("== A. single condition 'right', guidance weight w ==")
ref = real(RIGHT)
print(" w   frac_right  mean|x|  std_x   precision recall")
for w in (0, 1, 2, 4, 8):
    f = lambda x, a: eps_hat(x, a, ALL) + w * (eps_hat(x, a, RIGHT) - eps_hat(x, a, ALL))
    g = sample(f)
    p, r = knn_pr(ref, g[:1500])
    print(f"{w:2d}   {np.mean(g[:,0]>0):9.3f}  {np.abs(g[:,0]).mean():7.3f}  {g[g[:,0]>0,0].std():6.3f}  {p:9.3f} {r:6.3f}")

print("\n== B. two conditions: composed vs joint ==")
for name, f in {
    "joint cond (right,top), w=1": lambda x, a: eps_hat(x, a, BOTH),
    "composed w_a=w_b=1": lambda x, a: eps_hat(x, a, ALL) + (eps_hat(x, a, RIGHT) - eps_hat(x, a, ALL)) + (eps_hat(x, a, TOP) - eps_hat(x, a, ALL)),
    "composed w_a=w_b=3": lambda x, a: eps_hat(x, a, ALL) + 3 * (eps_hat(x, a, RIGHT) - eps_hat(x, a, ALL)) + 3 * (eps_hat(x, a, TOP) - eps_hat(x, a, ALL)),
    "only right, w=1": lambda x, a: eps_hat(x, a, RIGHT),
}.items():
    g = sample(f)
    print(f"{name:30s}", quad(g))

print("\n== C. negative condition: want 'right', negative 'top' ==")
for w in (1, 2, 4):
    f = lambda x, a: eps_hat(x, a, TOP) + w * (eps_hat(x, a, RIGHT) - eps_hat(x, a, TOP))
    g = sample(f)
    print(f"w={w}: ", quad(g))

Program output (NumPy 2.5.3, Python 3.12; 4,000 samples per row, seed fixed):

== A. single condition 'right', guidance weight w ==
 w   frac_right  mean|x|  std_x   precision recall
 0       0.490    1.990   0.286      0.467  0.967
 1       1.000    1.999   0.289      0.977  0.976
 2       1.000    2.193   0.207      0.983  0.842
 4       1.000    2.410   0.168      0.986  0.485
 8       1.000    2.689   0.147      0.977  0.140

== B. two conditions: composed vs joint ==
joint cond (right,top), w=1    {'right-top': 1.0, 'right-bottom': 0.0, 'left-top': 0.0, 'left-bottom': 0.0}
composed w_a=w_b=1             {'right-top': 1.0, 'right-bottom': 0.0, 'left-top': 0.0, 'left-bottom': 0.0}
composed w_a=w_b=3             {'right-top': 1.0, 'right-bottom': 0.0, 'left-top': 0.0, 'left-bottom': 0.0}
only right, w=1                {'right-top': 0.494, 'right-bottom': 0.506, 'left-top': 0.0, 'left-bottom': 0.0}

== C. negative condition: want 'right', negative 'top' ==
w=1:  {'right-top': 0.494, 'right-bottom': 0.506, 'left-top': 0.0, 'left-bottom': 0.0}
w=2:  {'right-top': 0.178, 'right-bottom': 0.822, 'left-top': 0.0, 'left-bottom': 0.0}
w=4:  {'right-top': 0.02, 'right-bottom': 0.98, 'left-top': 0.0, 'left-bottom': 0.0}

Part A sweeps the guidance scale for the single condition “right”. Precision and recall here are measured against 2,000 fresh points drawn from the “right” clusters. At \(w=0\) the model is unconditional: 49% of samples land on the right, and precision against the right-only reference is 0.467, the expected half. At \(w=1\), the ordinary conditional prediction, every sample is on the right and precision and recall are both near 0.98. In this toy the conditional model is exact, so \(w=1\) already delivers full adherence. Raising \(w\) further pushes the samples outward: the mean absolute x grows from 2.0 to 2.69, past the cluster centers at 2, the within-side spread shrinks from 0.29 to 0.15, and recall falls from 0.976 to 0.140. The samples are all valid-looking but cover less of the reference distribution. Real models are different in one important way: their learned conditional predictions are imperfect, so a scale above 1 can repair weak adherence, and the price is the same loss of coverage and drift beyond the data’s range that the toy shows in pure form.

Part B compares joint conditioning on both attributes with the composed formula. With \(w_1=w_2=1\), composition reproduces the joint prediction exactly (every sample in the top-right quadrant), because the toy’s conditions are independent given the point by construction. Scaling both weights to 3 still lands in the right quadrant, since the data has only one top-right cluster. Conditioning on “right” alone splits the samples between top-right and bottom-right, 49% and 51%. Part C uses the negative formulation. With “right” as the target and “top” as the negative condition, \(w=1\) recovers the 50/50 split, \(w=2\) moves 82% of samples to bottom-right and \(w=4\) moves 98%. Pushing away from one attribute works smoothly here. In a learned text-to-image model, the negative prompt’s implicit classifier is only as good as the model’s grasp of the negated concept, and an unwanted concept the model entangles with the wanted one will be pulled along.

Each additional condition costs one more forward pass per step in the straightforward implementation (a joint conditional pass, one per separate condition, and an unconditional or negative pass). Implementations batch these passes, so the wall-clock cost grows less than the pass count, but the compute budget does grow, and that is one reason systems often fold several controls into a single conditioning input.

Spatial and image conditions

Text says what should appear; it is a poor tool for saying where, how large, or in what pose. Spatial conditions fill that gap, and there are three common ways to supply them.

Starting from a source image

The cheapest image condition needs no new training. To edit or restyle a source image, add noise to its latent up to an intermediate level and run the sampler from there. The forward formula gives the mixture directly: \(x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\varepsilon\). Tools expose the starting point as a strength between 0 and 1, where 1 means starting from pure noise; the idea is the one used by SDEdit (Meng et al.). The coefficients show what the strength setting means.

import numpy as np
from common import ab_fn

print("strength  alpha_bar  source_coef  noise_coef")
for s in (0.2, 0.4, 0.6, 0.8, 1.0):
    a = ab_fn(min(s, 0.999))          # strength s means: start denoising from noise level u = s
    print(f"{s:8.1f}  {a:9.4f}  {np.sqrt(a):11.4f}  {np.sqrt(1 - a):10.4f}")

strength  alpha_bar  source_coef  noise_coef
     0.2     0.8987       0.9480      0.3183
     0.4     0.6475       0.8047      0.5937
     0.6     0.3408       0.5838      0.8119
     0.8     0.0940       0.3067      0.9518
     1.0     0.0000       0.0016      1.0000

These are the cosine-schedule values used throughout. At strength 0.4 the source keeps 80% of its amplitude and the noise coefficient is 0.59, so global layout and large shapes survive and fine detail is regenerated. At 0.8 the source coefficient is 0.31 against a noise coefficient of 0.95: only the coarsest color layout remains. At 1.0 the source coefficient is 0.0016 and the image is gone. The trade-off is built into the method: a low strength preserves identity and limits edits, a high strength allows large edits and loses identity. No setting preserves the structure of a face and changes its expression independently, which is the reason for the next two mechanisms.

Pixel-aligned inputs and trainable side branches

Inpainting and super-resolution are typically trained with the condition concatenated to the input channels. The training pairs are easy to synthesize: take any image, mask a region, and ask the model to restore it. Edges, depth maps and poses are harder to bake into the base model one by one, and retraining a large model for each control is expensive. Zhang et al. propose ControlNet, which keeps the pretrained model locked, makes a trainable copy of its encoder blocks that reads the condition, and connects the copy to the locked network through convolutions initialized to zero.

The zero initialization has a specific purpose. A new branch with random weights would inject noise into a finely tuned model at the first step and could damage it. With zero weights the branch’s contribution is exactly zero, so the combined network starts as the original model. The question is whether zero weights can learn at all, since a layer whose output is zero might seem to block gradients. The next listing checks it on a linear layer, which is what a 1×1 convolution is.

import numpy as np

rng = np.random.default_rng(0)
x = rng.normal(size=(4, 8))          # control features entering the projection
W = np.zeros((8, 8))                 # zero-initialized projection (a 1x1 convolution is a matrix product)
y = x @ W                            # what gets added to the frozen network's features
g_up = rng.normal(size=y.shape)      # gradient arriving from the loss
print("output at initialization, max abs:        ", np.abs(y).max())
print("gradient wrt W, max abs:                   ", round(float(np.abs(x.T @ g_up).max()), 3))
print("gradient flowing back into x, max abs:     ", np.abs(g_up @ W.T).max())

output at initialization, max abs:         0.0
gradient wrt W, max abs:                    5.181
gradient flowing back into x, max abs:      0.0

The output at initialization is exactly zero, so the base model is undisturbed. The gradient with respect to the weights is not zero (largest entry 5.18), because it is the product of the layer’s input and the incoming gradient, and neither is zero. The weights therefore move on the first update. The gradient flowing back into the control features is zero at initialization, because it passes through the zero weights. The branch itself begins to receive gradient one step later, once the weights are nonzero. Zero initialization delays learning inside the branch by a step and does not prevent it.

A second family conditions on images rather than on pixel-aligned maps. Ye et al. propose IP-Adapter, which adds a separate cross-attention path for features of a reference image, alongside the existing text cross-attention, so that image and text prompts can be used together. Because a reference image can carry a person’s face or an artist’s style, this kind of conditioning enlarges the misuse surface discussed later. Personalization by fine-tuning on a handful of photographs, for example with low-rank adapters of the kind described in Fine-Tuning and Alignment: SFT, LoRA, RLHF, and DPO, has the same property.

Each of these controls makes a promise that evaluation should test: that the output follows the supplied structure. A depth-conditioned image can look good and ignore the depth map. Adherence to a spatial condition is measured by extracting the same signal from the output (running a depth estimator or edge detector on the result) and comparing it with the input condition, which makes the quality of the estimator part of the measurement.

From images to video

A video is a tensor of shape (frames, height, width, channels). The sampler, the schedule and the noise-prediction loss carry over unchanged; the denoiser has to handle the extra axis. Three things change: the amount of data per sample, the dependence between frames that the model must represent, and the conditions that make sense.

Latent video and the token count

Pixel-space diffusion over many frames is impractical, so video models usually work in a compressed latent space, as latent image models do. A video autoencoder, trained in the manner of the variational autoencoders described earlier, compresses space and often time. Transformer denoisers cut the latent into patches, as in Vision Transformers and Modern Vision Models, and treat every patch of every latent frame as a token. Full self-attention over all tokens is quadratic in their number. A cheaper alternative factorizes attention: spatial attention inside each frame, then temporal attention across frames at each spatial location.

Consider a hypothetical configuration with 8× spatial compression, 4× temporal compression (the first frame kept alone, then one latent frame per four frames) and 2×2 patches. Compression factors differ between published models; these are chosen only to make the arithmetic concrete.

def counts(frames, height, width, t_comp=4, s_comp=8, patch=2):
    t = 1 + (frames - 1) // t_comp                    # latent frames after temporal compression
    h, w = height // s_comp // patch, width // s_comp // patch
    n_sp = h * w                                      # tokens per latent frame
    full = (t * n_sp) ** 2                            # query-key pairs, full space-time attention
    factorized = t * n_sp ** 2 + n_sp * t ** 2        # spatial attention per frame + temporal attention per location
    return t, n_sp, t * n_sp, full, factorized

print("frames  size        latent_frames  tokens/frame  total_tokens  full_pairs  factorized_pairs  ratio")
for frames, height, width in [(1, 512, 512), (17, 512, 512), (49, 768, 768), (121, 1280, 720)]:
    t, n_sp, tot, full, fact = counts(frames, height, width)
    print(f"{frames:6d}  {height}x{width:<6d} {t:13d}  {n_sp:12d}  {tot:12d}  {full:.3e}  {fact:.3e}  {full / fact:6.1f}")

frames  size        latent_frames  tokens/frame  total_tokens  full_pairs  factorized_pairs  ratio
     1  512x512                1          1024          1024  1.049e+06  1.050e+06     1.0
    17  512x512                5          1024          5120  2.621e+07  5.268e+06     5.0
    49  768x768               13          2304         29952  8.971e+08  6.940e+07    12.9
   121  1280x720               31          3600        111600  1.245e+10  4.052e+08    30.7

A single 512×512 image gives 1,024 tokens, and factorized attention has no advantage over full attention. At 17 frames the clip has 5 latent frames and 5,120 tokens, and the factorized form needs five times fewer query-key pairs. A five-second clip at 24 frames per second and 1280×720 pixels (121 frames) has 31 latent frames and 111,600 tokens, so full attention needs about 1.2×10¹⁰ pairs per attention layer per head, and the factorized form 4.1×10⁸, about 31 times fewer. Memory-efficient kernels such as the one discussed in LLM Inference Optimization: KV Cache, FlashAttention, and Quantization avoid storing the full attention matrix but not the arithmetic. These counts cover one layer; the denoiser is applied at every sampling step, with extra passes for guidance, so video generation latency is dominated by this term.

The saving has a cost. In factorized attention a token does not attend directly to a token in another frame at a different location. Information about a moving object has to travel through a spatial layer and then a temporal layer, which limits how fast and how far the model can propagate motion within a block. Full space-time attention removes that limit at the quadratic cost above.

Four ways to build the denoiser

• Inflate an image model. Start from a pretrained image denoiser and insert temporal layers (temporal attention or 1D or 3D convolutions) between its spatial layers. Ho et al.’s Video Diffusion Models factorize attention over space and time and train jointly on images and videos, so the same network sees single frames as videos of length one. Stable Video Diffusion (Blattmann et al.) reports three stages: text-to-image pretraining, video pretraining on a curated dataset, and fine-tuning on a smaller high-quality set. The inherited image prior is the main advantage; the main limit is that spatial layers were trained with no notion of time.
• A cascade. Imagen Video (Ho et al.) generates a low-resolution, low-frame-rate base video and then refines it with interleaved spatial and temporal super-resolution models. Each stage solves a smaller problem, and each can be conditioned on the output of the previous one. Errors made by the base model are inherited by every later stage.
• A joint space-time Transformer. Patchify the whole latent video and run full or hybrid attention over all tokens. This gives the model the most freedom and the highest cost, and it relies on a good video autoencoder.
• Autoregressive extension. Generate a short clip, then generate the next clip conditioned on the last frames of the previous one. This reaches arbitrary length at fixed memory, and it accumulates error, as discussed below.

Conditions specific to video

Text still enters through cross-attention or modulation, and the pathways in the first table apply. Video adds conditions defined along time. An image-to-video model fixes the first frame: the clean first-frame latent is supplied as extra input channels or substituted for the noisy first frame at every step. Keyframe conditioning fixes several frames and asks the model to fill in between. Camera conditioning supplies a per-frame camera pose as a sequence of vectors. Motion conditioning supplies trajectories, optical flow, or a driving video. Audio conditioning, for talking heads and music-driven clips, requires audio features at the video’s frame rate. The audio side of that pipeline, framing, spectral features and the latency of streaming audio, is the subject of Speech Systems: Recognition, Generation, and Streaming. The video-side requirement is only that the audio feature sequence be resampled to one vector (or a fixed number of vectors) per latent frame, which is a choice about alignment: a feature rate that does not divide the frame rate has to be interpolated.

Temporal consistency

Temporal consistency has four distinct meanings, and they fail differently. Appearance consistency means colors, textures and lighting stay stable; its failure is flicker. Identity consistency means the same object keeps the same shape and attributes; its failure is an object morphing or a face changing. Geometric and physical consistency means objects move plausibly and obey occlusion; its failure is limbs passing through bodies or objects vanishing. Motion adequacy means the clip actually moves; its failure is a nearly static video, which is trivially consistent and useless.

Why independent frames flicker

Suppose each frame were generated by a diffusion model that treats frames as independent. Every frame is drawn from the correct per-frame distribution, and the pairs of adjacent frames are drawn from the product of two marginals instead of the joint distribution. If the true distribution says that adjacent frames are close, the product distribution says they are not, so the clip jumps by the typical difference between two random frames. This is not a flaw in the sampler; it follows from what the denoiser is allowed to see.

A Gaussian toy shows this without training a network. Let a "clip" be 16 scalar frame values drawn from a zero-mean Gaussian with covariance \(\Sigma_{ij}=\rho^{|i-j|}\) and \(\rho=0.9\), so neighboring frames correlate at 0.9 and each frame has unit variance. For Gaussian data the ideal denoiser is linear: \(E[x_0\mid x_t]=\sqrt{\bar\alpha_t}\,\Sigma(\bar\alpha_t\Sigma+(1-\bar\alpha_t)I)^{-1}x_t\). The joint denoiser uses the full \(\Sigma\). The per-frame denoiser uses \(\Sigma=I\), which has the same per-frame variance and no cross-frame structure, standing in for a network whose layers cannot exchange information between frames. Both are sampled with the same 50-step deterministic DDIM update. Four variants are compared: the joint denoiser (A), the per-frame denoiser with independent noise in each frame (B), the per-frame denoiser with the same starting noise value in all frames (C), and variant B followed by a three-frame moving average, the post-hoc fix that is often tried first (D).

import numpy as np
from common import ab_fn

F, RHO, N = 16, 0.9, 3000
idx = np.arange(F)
SIG_JOINT = RHO ** np.abs(idx[:, None] - idx[None])       # frames correlated: AR(1) covariance
SIG_INDEP = np.eye(F)                                      # same per-frame variance, no cross-frame dependence

def denoiser(Sigma):
    def eps(x, ab):
        M = ab * Sigma + (1 - ab) * np.eye(F)
        x0 = np.sqrt(ab) * np.linalg.solve(M, x.T).T @ Sigma.T     # E[x0 | x_t] for Gaussian data
        return (x - np.sqrt(ab) * x0) / np.sqrt(1 - ab)
    return eps

def ddim(eps_fn, x, steps=50):
    us = np.linspace(0.999, 0.0, steps + 1)
    for i in range(steps):
        a_t, a_p = ab_fn(us[i]), ab_fn(us[i + 1])
        e = eps_fn(x, a_t)
        x0 = (x - np.sqrt(1 - a_t) * e) / np.sqrt(a_t)
        x = np.sqrt(a_p) * x0 + np.sqrt(1 - a_p) * e
    return x

def stats(name, X):
    d = np.abs(np.diff(X, axis=1)).mean()
    c = np.mean([np.corrcoef(X[:, i], X[:, i + 1])[0, 1] for i in range(F - 1)])
    var_frame = X.var(0).mean()
    motion = X.std(1).mean()          # spread of values over time inside one clip
    print(f"{name:34s} mean|dx| {d:6.3f}  lag1 corr {c:6.3f}  per-frame var {var_frame:6.3f}  within-clip std {motion:6.3f}")

rng = np.random.default_rng(0)
L = np.linalg.cholesky(SIG_JOINT)
stats("training data (AR(1), rho=0.9)", (L @ rng.normal(size=(F, N))).T)
z = rng.normal(size=(N, F))
stats("A joint denoiser", ddim(denoiser(SIG_JOINT), z.copy()))
stats("B per-frame denoiser, indep. noise", ddim(denoiser(SIG_INDEP), z.copy()))
zs = np.repeat(rng.normal(size=(N, 1)), F, axis=1)
stats("C per-frame denoiser, shared noise", ddim(denoiser(SIG_INDEP), zs))
B = ddim(denoiser(SIG_INDEP), z.copy())
smooth = np.stack([np.convolve(r, np.ones(3) / 3, mode="valid") for r in B])
Fs = smooth.shape[1]
d = np.abs(np.diff(smooth, axis=1)).mean()
c = np.mean([np.corrcoef(smooth[:, i], smooth[:, i + 1])[0, 1] for i in range(Fs - 1)])
print(f"{'D B + 3-frame moving average':34s} mean|dx| {d:6.3f}  lag1 corr {c:6.3f}  per-frame var {smooth.var(0).mean():6.3f}  within-clip std {smooth.std(1).mean():6.3f}")

Program output (NumPy 2.5.3, Python 3.12; 3,000 clips per row):

training data (AR(1), rho=0.9)     mean|dx|  0.357  lag1 corr  0.897  per-frame var  0.977  within-clip std  0.588
A joint denoiser                   mean|dx|  0.341  lag1 corr  0.900  per-frame var  0.915  within-clip std  0.568
B per-frame denoiser, indep. noise mean|dx|  1.101  lag1 corr -0.004  per-frame var  0.951  within-clip std  0.929
C per-frame denoiser, shared noise mean|dx|  0.000  lag1 corr  1.000  per-frame var  0.945  within-clip std  0.000
D B + 3-frame moving average       mean|dx|  0.367  lag1 corr  0.664  per-frame var  0.317  within-clip std  0.486

Read the columns as follows. Mean |dx| is the average absolute change between adjacent frames, the quantity a viewer perceives as jitter. Lag-1 correlation is the correlation across clips between adjacent frames. Per-frame variance checks that each frame still has the right spread (1.0 would be exact; the training data row shows the finite-sample estimate, 0.977). Within-clip std is how much values vary over time inside one clip, the proxy for motion.

• A matches the data: mean change 0.341 against 0.357, lag-1 correlation 0.900 against 0.897. Cross-frame dependence inside the denoiser is what produces temporally coherent samples.
• B has correlation near zero and jumps of 1.10, about three times the data’s. Each frame is individually plausible, and the clip flickers.
• C has zero jitter and correlation 1.0, which a consistency metric would score as perfect. It is also frozen: within-clip std is exactly zero. A per-frame denoiser maps equal inputs to equal outputs, so sharing the starting noise makes all frames identical. Consistency scored alone would reward this clip.
• D brings the jitter down to 0.367, close to the data’s 0.357, but the lag-1 correlation is only 0.664 and the per-frame variance falls to 0.317. Averaging three independent frames shrinks the variance by about a factor of three. In an image or video this is blur: temporal smoothing removes flicker by removing the detail that varied. Matching one statistic hides the damage done to another.

In a real model the conditions are rarely this clean, but the same pattern holds. Weak cross-frame coupling gives flicker. Over-strong coupling, for example from a heavy shared conditioning signal, gives stillness. Post-processing trades one defect for another. That is the reason the evaluation section pairs a consistency measure with a motion measure and a per-frame quality measure.

Long videos and drift

Autoregressive extension and sliding windows cause drift for a simple reason: each new chunk is conditioned on frames that the model itself generated, which already contain small errors, and the model was trained mostly on clean conditioning frames. Errors compound the way they do in any autoregressive rollout. Common mitigations include corrupting the conditioning frames with noise during training so the model learns to tolerate imperfect context, overlapping chunks and blending in the overlap, generating sparse keyframes first and filling in between them, and reconditioning on a fixed reference frame to anchor identity. Each has a cost. Overlap increases compute, keyframe-first generation limits fast motion between keyframes, and a fixed anchor makes it harder for the scene to change.

Evaluating generated images and video

No single number says that a generator is good. Evaluation splits into questions with different tools, and a report that gives one score for all of them hides which question the score answers.

Question	Typical measurement	What it misses
Do samples look like real data? (fidelity)	Precision-style metrics, classifier or human realism judgments	Whether the model covers the data
Does the model cover the variety of the data? (diversity, coverage)	Recall-style metrics, distribution distances such as FID	Individual-sample quality
Does the output match the prompt?	Embedding similarity, detector- or question-based checks, human rating	Aesthetic quality; the checker’s own errors
Do people prefer it?	Pairwise human preference, fitted to a ranking	Cost, narrow prompt coverage, annotator bias
Is it stable over time? (video)	Frame-to-frame feature similarity, warping error, flicker measures	Motion adequacy; a frozen video scores perfectly
Does it move enough and plausibly? (video)	Motion magnitude statistics, human review of physics and identity	Per-frame quality
Is it safe and attributable?	Red-team prompt sets, classifier miss rates, watermark detection rates	Attacks not in the test set

Distribution metrics and what they hide

The Fréchet Inception Distance of Heusel et al. embeds real and generated images with a fixed pretrained network, fits a Gaussian to each set of features, and computes the Fréchet distance between the two Gaussians. For means \(\mu_r,\mu_g\) and covariances \(\Sigma_r,\Sigma_g\):

\[\mathrm{FD}=\lVert\mu_r-\mu_g\rVert^2+\mathrm{Tr}\!\left(\Sigma_r+\Sigma_g-2\,(\Sigma_r\Sigma_g)^{1/2}\right)\]

The first term penalizes a shift in the mean feature; the second penalizes a mismatch in the spread and orientation of the features. FD is zero when the two Gaussians coincide. Several properties matter for how the number is used. It depends on the feature extractor, so scores from different extractors cannot be compared. It assumes the features are Gaussian, which they are not. Preprocessing (resizing and compression) changes the features and so changes the score. And the estimate is biased by the number of samples, as the first experiment below shows. Kernel inception distance (Bińkowski et al.) replaces the Gaussian fit with an unbiased estimate of a kernel distance between the feature sets; Jayasumana et al. argue for a CLIP-feature variant (CMMD) for text-to-image evaluation on the grounds that Inception features and the Gaussian assumption are poor fits for modern models.

The experiments use synthetic features, not images, so every property can be checked against a known truth. Part A draws two sets from the same 64-dimensional Gaussian, where the true distance is zero, and reports the average over five repeats. Part B uses 16-dimensional features from a mixture of four well-separated modes and compares the real set with four generated sets: one from the same distribution, one missing a mode, one "collapsed" onto the mode centers (the high-fidelity, low-diversity pattern that strong guidance produces), and one noisier than the data. FD is shown next to the k-nearest-neighbor precision and recall of Kynkäänniemi et al.: a generated point counts toward precision if it falls inside the ball around a real point whose radius is the distance to that point’s k-th real neighbor, and a real point counts toward recall if it falls inside such a ball around a generated point. Precision reads as realism and recall as coverage. Part C belongs to the statistics discussion further below.

import numpy as np
from common import knn_pr

def fd(a, b):
    mu1, mu2 = a.mean(0), b.mean(0)
    s1, s2 = np.cov(a, rowvar=False), np.cov(b, rowvar=False)
    ev = np.linalg.eigvals(s1 @ s2).real.clip(0)           # eigenvalues of S1 S2 are real and nonnegative
    return float(((mu1 - mu2) ** 2).sum() + np.trace(s1) + np.trace(s2) - 2 * np.sqrt(ev).sum())

def kid(a, b):
    d = a.shape[1]
    k = lambda x, y: (x @ y.T / d + 1) ** 3
    n, m = len(a), len(b)
    kxx, kyy, kxy = k(a, a), k(b, b), k(a, b)
    return float((kxx.sum() - np.trace(kxx)) / (n * (n - 1)) + (kyy.sum() - np.trace(kyy)) / (m * (m - 1)) - 2 * kxy.mean())

rng = np.random.default_rng(0)
print("== A. sample-size bias, both sets drawn from the SAME 64-d Gaussian ==")
print("   N     FD(mean of 5)   KID(mean of 5)")
for n in (64, 256, 1024, 4096):
    f_, k_ = [], []
    for _ in range(5):
        a, b = rng.normal(size=(n, 64)), rng.normal(size=(n, 64))
        f_.append(fd(a, b)); k_.append(kid(a[:min(n, 1024)], b[:min(n, 1024)]))
    print(f"{n:5d}   {np.mean(f_):12.3f}   {np.mean(k_):13.5f}")

print("\n== B. one number versus two: synthetic 16-d features, 4 modes ==")
D = 16
centers = rng.normal(size=(4, D)) * 3.0
def draw(n, scale=1.0, modes=(0, 1, 2, 3), shrink=1.0):
    c = centers[rng.choice(modes, n)]
    return c * 1.0 + rng.normal(size=(n, D)) * scale * shrink
real = draw(3000)
variants = {
    "same distribution": draw(3000),
    "dropped one mode": draw(3000, modes=(0, 1, 2)),
    "collapsed (std x0.25)": draw(3000, shrink=0.25),
    "noisy (std x2.0)": draw(3000, shrink=2.0),
}
print(f"{'generated set':24s} {'FD':>8s} {'precision':>10s} {'recall':>8s}")
for name, g in variants.items():
    p, r = knn_pr(real[:1500], g[:1500], k=5)
    print(f"{name:24s} {fd(real, g):8.2f} {p:10.3f} {r:8.3f}")

print("\n== C. prompt-level checker, simulated pass/fail, paired bootstrap over prompts ==")
sig = lambda z: 1 / (1 + np.exp(-z))
def trial(P, S):
    base = rng.normal(0.0, 1.5, size=P)                     # prompt difficulty (logit)
    gain = 0.2 + rng.normal(0.0, 0.8, size=P)               # model B helps some prompts more than others
    A = (rng.random((P, S)) < sig(base[:, None])).astype(float)
    B = (rng.random((P, S)) < sig((base + gain)[:, None])).astype(float)
    dpp = B.mean(1) - A.mean(1)                             # one paired difference per prompt
    boots = np.array([dpp[rng.integers(0, P, P)].mean() for _ in range(4000)])
    lo, hi = np.percentile(boots, [2.5, 97.5])
    print(f"prompts {P:4d}  seeds {S:2d}  diff {dpp.mean():+.3f}  95% CI [{lo:+.3f}, {hi:+.3f}]  half-width {(hi-lo)/2:.3f}")
for P, S in [(50, 4), (200, 4), (800, 4), (200, 1), (200, 16)]:
    trial(P, S)

Program output (NumPy 2.5.3, Python 3.12; synthetic features, fixed seed):

== A. sample-size bias, both sets drawn from the SAME 64-d Gaussian ==
   N     FD(mean of 5)   KID(mean of 5)
   64         34.985        -0.00538
  256          8.841         0.00052
 1024          2.154        -0.00004
 4096          0.539         0.00018

== B. one number versus two: synthetic 16-d features, 4 modes ==
generated set                  FD  precision   recall
same distribution            0.35      0.894    0.858
dropped one mode            35.60      0.869    0.656
collapsed (std x0.25)        7.40      1.000    0.000
noisy (std x2.0)            14.08      0.040    1.000

Part A: with true distance zero, FD reads 35.0 at 64 samples per set, 8.8 at 256, 2.2 at 1,024 and 0.54 at 4,096. The bias is positive and shrinks with the sample size; at the same sample size it grows with feature dimension (here 64, against the 2,048 of Inception features), because there are more covariance entries to estimate. Two consequences follow: FD values are comparable only at the same sample count and extractor, and small differences between models at small sample counts carry no information. The unbiased kernel estimate averages around zero (values between −0.0054 and 0.0005, with both signs), as expected from an unbiased estimator.

Part B: the same-distribution set has FD 0.35, precision 0.894 and recall 0.858. Precision and recall are below 1 even for the true distribution because k-nearest-neighbor balls estimated from finite samples in 16 dimensions miss part of the distribution; the right comparison is with this baseline, not with 1. Dropping a mode leaves precision at the baseline (0.869) and cuts recall to 0.656, roughly the baseline times three quarters, and FD jumps to 35.6. The collapsed set gets precision 1.000 and recall 0.000: every sample is a good-looking point near a center and almost none of the real points are covered. The noisy set gets precision 0.040 and recall 1.000. The FD values for the two failures, 7.4 and 14.1, are both smaller than the mode-drop value of 35.6, and the single number says nothing about which failure occurred. On other data the ordering could differ. The precision-recall pair separates them in this experiment, and the pair is what guidance sweeps should report: raising the guidance scale moves a text-to-image model along a frontier from the diverse and imprecise end towards the precise and collapsed end, as the toy in the guidance section did.

Prompt alignment

CLIPScore (Hessel et al.) computes the cosine similarity between a CLIP image embedding and a CLIP text embedding of the prompt, clipped at zero and rescaled. It is cheap and correlates with human judgments of caption quality. A single-vector similarity has no explicit slot for which attribute belongs to which object, so a prompt and an image with swapped attributes can receive similar scores. Counting, relative position and attribute binding are the failure modes that conditioning mechanisms are most prone to, so measuring them needs a step that checks structure.

Ghosh et al.’s GenEval does this with an object detector: for each prompt it checks object co-occurrence, count, color and position against the detected objects, so the score is a pass rate per skill. Question-answering approaches decompose the prompt into questions (“is there a cube?”, “is the cube red?”, “is the cube on the sphere?”), answer them with a vision-language model, and aggregate. Both are only as good as the detector or answering model, so the checker needs its own accuracy audit on a labeled sample, and a score above the checker’s accuracy ceiling cannot be trusted. The mechanics are those of any classifier evaluation; Classification Metrics and Decision Thresholds covers them.

Human preference

Pairwise comparison is the most common human protocol: show two images for the same prompt, ask which is better on a stated criterion (prompt adherence, realism, aesthetics), and fit a ranking. The Bradley–Terry model gives model \(i\) a strength \(\theta_i\) and sets the probability that \(i\) beats \(j\) to \(\sigma(\theta_i-\theta_j)\). A strength gap of 0.4 corresponds to a win probability of about 0.60, which is a clear but modest preference. Ask separate questions for separate criteria; one overall "better" question mixes them and is dominated by whichever the annotators weigh most. Report agreement between annotators, randomize order, and keep annotator instructions fixed across model versions.

Statistics for prompt-based comparisons

Pass rates from a checker are averages over a prompt set. A model that differs from another by two points on 200 prompts may be no better at all. Part C of the program simulates the situation: each prompt has a difficulty, two models pass with probabilities that depend on it, one model has a small average advantage that varies by prompt, and each prompt is sampled with several seeds. The comparison is paired, so there is one difference per prompt (the mean over seeds), and the bootstrap resamples prompts, because seeds of the same prompt are not independent samples of the prompt population. The bootstrap method itself and the paired design are discussed in Comparing Models: Uncertainty and Significance.

== C. prompt-level checker, simulated pass/fail, paired bootstrap over prompts ==
prompts   50  seeds  4  diff +0.115  95% CI [+0.040, +0.190]  half-width 0.075
prompts  200  seeds  4  diff +0.040  95% CI [-0.001, +0.083]  half-width 0.042
prompts  800  seeds  4  diff +0.022  95% CI [-0.000, +0.044]  half-width 0.022
prompts  200  seeds  1  diff +0.000  95% CI [-0.075, +0.075]  half-width 0.075
prompts  200  seeds 16  diff +0.006  95% CI [-0.023, +0.034]  half-width 0.029

Each row is a separate simulated experiment, so the point estimates differ between rows; the interval widths are the point. With 50 prompts the half-width of the 95% interval is 0.075; with 200 it is 0.042, and with 800 it is 0.022, roughly halving each time the prompt set quadruples. At 200 prompts and 4 seeds the interval just reaches zero (−0.001 to +0.083). Seeds help less than prompts: at 200 prompts, going from 1 seed to 4 to 16 narrows the half-width from 0.075 to 0.042 to 0.029, and the remaining width comes from differences between prompts, which more seeds cannot reduce. When resources are limited, add prompts before adding seeds, and report the interval together with the difference. The data here are simulated pass/fail outcomes, not measurements of real models.

Video metrics

Fréchet Video Distance (Unterthiner et al.) is FD computed on features from a video network. Ge et al. report that FVD is biased toward the quality of individual frames over temporal consistency, which makes it a weak measure of the property video models are hardest at and suggests pairing it with a metric that targets time. Benchmark suites such as VBench (Huang et al.) split video quality into 16 dimensions, including motion smoothness, temporal flickering and spatial relationships, so a model’s strengths and weaknesses can be read separately. Common time-focused measures are feature similarity between adjacent frames in a self-supervised embedding space (stable appearance and identity), warping error (estimate optical flow from frame t to t+1, warp the first frame, and compare it with the second, which penalizes flicker and is distorted by flow errors and occlusion), and a motion statistic such as mean flow magnitude. The toy in the temporal-consistency section is the reason for the last one: variant C would win every stability metric.

An evaluation protocol

1. Fix the prompt set before comparing models, include categories that stress conditioning (counting, attributes, spatial relations, negation, long prompts) and keep a held-out subset that nobody tunes against.
2. Fix the sampler, step count, resolution, guidance scale, seeds and preprocessing, and report them. For guidance, report a sweep (precision and recall, or FID and a prompt-alignment score against the scale) instead of one point.
3. Use the same sample count for every distribution metric and a reference set that matches the target domain.
4. Pair every consistency metric with a motion metric and a per-frame quality metric.
5. Audit automated checkers on a labeled sample and report their error rates next to the scores.
6. Report intervals for differences, with prompts as the resampling unit, and confirm automated rankings against a small human preference study before acting on small gaps.
7. Screen outputs for near-copies of training images (see the safety section) so that fidelity is not memorization.

Provenance

The question of provenance is: given an image or clip, can anyone find out whether it was generated, by what, and with which edits? Three mechanisms address it, and they have different guarantees.

Signed manifests

The Coalition for Content Provenance and Authenticity (C2PA) specification defines a manifest: a set of assertions about an asset (for example, that a generator created it, or that an edit was applied) bound into a single claim that carries a cryptographic signature. The specification distinguishes two ways of tying the manifest to the asset. A hard binding uses a cryptographic hash of the asset’s bytes, so a validator can check that the manifest belongs to this asset and that the asset is unmodified. A soft binding uses a fingerprint or an invisible watermark, which lets a validator find the manifest even when the bits of the asset differ, for instance after re-encoding.

Invisible watermarks

A watermark hides a signal in the pixels (or in the generation process) that survives some transformations and can be detected with a secret key. A minimal scheme adds a key-seeded pseudorandom pattern to the image at low amplitude and detects it by correlation. The detector removes the image’s own content with a local mean (otherwise the host’s structure swamps the pattern), correlates the residual with the key’s pattern, and normalizes so that the statistic is approximately standard normal for an unmarked image. A threshold on that statistic sets the false-positive rate: z above 4 corresponds to a one-sided rate of about 3×10⁻⁵ if the null is normal.

The listing compares two patterns of equal energy (so equal PSNR): one at pixel scale and one smoothed so that its structure is coarser. It marks 60 synthetic images (random low-frequency cosines plus mild texture, not photographs) and applies five attacks: Gaussian noise, a 5×5 blur, a JPEG-like 8×8 DCT quantization (a uniform step of 80 on the coefficients, simpler than real JPEG), a 2× downscale and upscale, and a 20-pixel horizontal shift. The listing ends with a stand-in for a signed manifest: an HMAC replaces the certificate-based signature of the real standard, so it shows only how a hash binding behaves and is not the C2PA format.

import hashlib, hmac, json
import numpy as np

rng = np.random.default_rng(7)
N = 128

def host(rng):
    """synthetic smooth 'photo': a few random low-frequency cosines plus mild texture, in [0,255]"""
    yy, xx = np.mgrid[0:N, 0:N] / N
    img = np.zeros((N, N))
    for _ in range(8):
        fx, fy, ph, amp = rng.integers(1, 5, 2).tolist() + [rng.uniform(0, 6.28), rng.uniform(10, 30)]
        img += amp * np.cos(2 * np.pi * (fx * xx + fy * yy) + ph)
    return np.clip(128 + img + rng.normal(0, 3, (N, N)), 0, 255)

def box(x, k):
    pad = k // 2
    xp = np.pad(x, pad, mode="edge")
    out = np.zeros_like(x)
    for dy in range(k):
        for dx in range(k):
            out += xp[dy:dy + N, dx:dx + N]
    return out / (k * k)

def unit_rms(w): return w / np.sqrt((w ** 2).mean())
key_rng = np.random.default_rng(12345)                       # the secret key seeds the pattern
W_HI = unit_rms(key_rng.choice([-1.0, 1.0], size=(N, N)))    # pixel-scale pattern
W_LO = unit_rms(box(key_rng.normal(size=(N, N)), 5))         # smoothed pattern (coarser structure)

def embed(img, w, alpha): return np.clip(img + alpha * w, 0, 255)

def zscore(img, w, k=5):
    r = img - box(img, k)                                    # remove host content with a local mean
    ww = w - box(w, k)
    s = (r * ww).sum()
    return s / np.sqrt((r ** 2).mean() * (ww ** 2).sum())    # approx N(0,1) when no watermark

def psnr(a, b): return 10 * np.log10(255 ** 2 / ((a - b) ** 2).mean())

def jpeg_like(img, q=40):
    n = 8
    C = np.array([[np.sqrt((1 if u == 0 else 2) / n) * np.cos((2 * x + 1) * u * np.pi / (2 * n)) for x in range(n)] for u in range(n)])
    out = np.zeros_like(img)
    for i in range(0, N, n):
        for j in range(0, N, n):
            blk = C @ (img[i:i+n, j:j+n] - 128) @ C.T
            out[i:i+n, j:j+n] = C.T @ (np.round(blk / q) * q) @ C + 128
    return np.clip(out, 0, 255)

def down_up(img): 
    s = img.reshape(N // 2, 2, N // 2, 2).mean((1, 3))
    return np.kron(s, np.ones((2, 2)))

attacks = {
    "none": lambda x: x,
    "gaussian noise sd=20": lambda x: x + np.random.default_rng(1).normal(0, 20, x.shape),
    "box blur 5x5": lambda x: box(x, 5),
    "JPEG-like quantization q=80": lambda x: jpeg_like(x, 80),
    "downscale x2 then upscale": down_up,
    "shift by 20 px": lambda x: np.roll(x, 20, axis=1),
}
hosts = [host(rng) for _ in range(60)]
ALPHA = 2.0
THRESH = 4.0
print("alpha", ALPHA, " mean PSNR (dB) of watermarked vs original:", round(float(np.mean([psnr(h, embed(h, W_HI, ALPHA)) for h in hosts])), 1))
for name, w in (("pixel-scale", W_HI), ("smoothed", W_LO)):
    null = np.array([zscore(h, w) for h in hosts])
    print(f"\n{name} pattern; z on unwatermarked images: mean {null.mean():+.2f} sd {null.std():.2f}")
    print(f"{'attack':28s} {'mean z':>8s} {'detected (z>4)':>15s}")
    for an, fn in attacks.items():
        z = np.array([zscore(fn(embed(h, w, ALPHA)), w) for h in hosts])
        print(f"{an:28s} {z.mean():8.1f} {np.mean(z > THRESH):15.2f}")

print("\n-- signed manifest stand-in (HMAC in place of an X.509/COSE signature) --")
SECRET = b"demo-signing-key"
def make_manifest(pixels: bytes):
    claim = {"generator": "toy-model-v0", "action": "created", "pixel_sha256": hashlib.sha256(pixels).hexdigest()}
    body = json.dumps(claim, sort_keys=True).encode()
    return {"claim": claim, "signature": hmac.new(SECRET, body, hashlib.sha256).hexdigest()}
def verify(manifest, pixels: bytes):
    body = json.dumps(manifest["claim"], sort_keys=True).encode()
    ok_sig = hmac.compare_digest(manifest["signature"], hmac.new(SECRET, body, hashlib.sha256).hexdigest())
    ok_bind = manifest["claim"]["pixel_sha256"] == hashlib.sha256(pixels).hexdigest()
    return ok_sig, ok_bind
img = np.round(host(rng)).astype(np.uint8)
m = make_manifest(img.tobytes())
print("original            (signature ok, hash matches):", verify(m, img.tobytes()))
reenc = np.round(jpeg_like(img.astype(float), q=8)).astype(np.uint8)
print("re-encoded copy     (signature ok, hash matches):", verify(m, reenc.tobytes()))
tampered = dict(m, claim=dict(m["claim"], action="captured by camera"))
print("edited claim text   (signature ok, hash matches):", verify(tampered, img.tobytes()))

Program output (NumPy 2.5.3, Python 3.12; synthetic images, fixed seeds):

alpha 2.0  mean PSNR (dB) of watermarked vs original: 42.1

pixel-scale pattern; z on unwatermarked images: mean -0.04 sd 1.03
attack                         mean z  detected (z>4)
none                             63.7            1.00
gaussian noise sd=20             13.3            1.00
box blur 5x5                     -2.6            0.00
JPEG-like quantization q=80       1.2            0.00
downscale x2 then upscale        13.4            1.00
shift by 20 px                    0.3            0.00

smoothed pattern; z on unwatermarked images: mean -0.12 sd 0.98
attack                         mean z  detected (z>4)
none                             38.5            1.00
gaussian noise sd=20              6.8            1.00
box blur 5x5                      7.9            1.00
JPEG-like quantization q=80       5.1            0.73
downscale x2 then upscale        17.6            1.00
shift by 20 px                    0.9            0.00

-- signed manifest stand-in (HMAC in place of an X.509/COSE signature) --
original            (signature ok, hash matches): (True, True)
re-encoded copy     (signature ok, hash matches): (True, False)
edited claim text   (signature ok, hash matches): (False, True)

The embedding amplitude gives a PSNR of 42.1 dB between the original and the marked image. On unmarked images the statistic has mean −0.04 and standard deviation 1.03 for the pixel pattern, close to the assumed standard normal, so the threshold of 4 is calibrated on 60 images; a deployed detector needs that rate measured on a large sample of natural images. Without attacks both patterns are detected in all images (mean z of 63.7 and 38.5). Under Gaussian noise both survive, since random noise is uncorrelated with the key. The pixel-scale pattern dies under the 5×5 blur (z = −2.6) and the quantization (z = 1.2), because both remove the fine structure where the pattern lives. The smoothed pattern survives the blur completely and the quantization in 73% of images (mean z 5.1, so some fall under 4). Both patterns fail under the shift, because the key’s pattern no longer lines up with the image; detection would need a search over alignments, or a design with built-in invariance. Tree-Ring watermarks (Wen et al.) take the second route: they put a structured pattern in the Fourier space of the initial noise of the diffusion process and detect it by inverting the sampler to recover that noise, which the authors report to be robust to several transformations such as cropping and rotation.

Several limits apply to every watermark scheme. Robustness and invisibility pull against each other: the smoothed pattern survives more here because it carries more energy at low frequencies, where the image cannot hide it as well and where compression keeps it. A watermark placed by the provider only covers that provider’s models, and open-weight models can be run without it. Anyone who knows the scheme can attempt removal or forgery, and a detector that misses a mark does not show that the image is real. The statement a detector supports is narrow: a mark of this key was found with this false-positive rate.

The last three lines of the output come from the manifest stand-in. The original image verifies on both checks: the signature is valid and the hash matches. A re-encoded copy keeps a valid signature but fails the hash binding, because the bytes changed, which is how a lossy re-save or a screenshot behaves. A manifest with an edited claim fails the signature check even though the pixels are unchanged. This is the practical weakness of hard bindings: they survive only as long as the bytes do, and a platform that strips metadata or re-compresses removes them without any attack. The soft binding exists to cover this gap, with the watermark or fingerprint used to look the manifest up again. A robust deployment therefore combines a signed manifest, which carries rich claims and is easy to strip, with a watermark, which carries little and is harder to strip, and treats the two outcomes as different statements.

Detectors and their limits

Classifiers trained to recognize generated images need no cooperation from the generator and can run on any image. They are trained on outputs of known generators and tend to degrade on new ones or after re-encoding, so their error rates depend on the generator mix at test time. Treat a detector score as evidence with a measured error rate, never as a verdict, and do not use it alone for decisions that affect a person.

Safety

Safety for image and video generators covers several different risks that share one product surface: deceptive or non-consensual imagery of real people, sexual content involving minors, graphic violence, impersonation and fraud, reproduction of protected or private training data, and stereotyped or exclusionary output. The product also decides whether a user may supply a reference image, which affects who can be impersonated. Legal duties around labeling, consent and content differ by jurisdiction and change over time, so the controls below are engineering choices that should be reviewed against the rules that apply to the deployment.

Layered controls

No single control is enough, and the controls act at different points.

• Training data. Filter unsafe content, remove duplicates, and document licensing and consent; what the model has never seen is the cheapest thing to keep out of its output, although filtering is imperfect.
• Prompt stage. Classify or rewrite the prompt, with attention to reference images and edit instructions as well as text. Cheap and early, and the easiest stage to evade by rewording.
• Model stage. Techniques that reduce a model’s ability to produce a concept, such as modifying the weights or steering the sampler away from unsafe directions with a guidance term of the kind described earlier. These are incomplete and can often be reversed by fine-tuning when the weights are available.
• Output stage. Classify the generated image or frames, plus a similarity check against known protected or sensitive images. This sees the actual result, at the cost of a generation already spent and of classifier errors.
• Product stage. Rate limits, account-level reputation, reporting channels, human review for borderline cases, and provenance marks on everything produced.

The value of stacking stages depends on whether their errors are independent. The final experiment simulates a prompt classifier and an output classifier on 200,000 requests, 1% of them harmful. Scores are Gaussian, and a harmful request has a mean score 2.5 higher than a benign one. In the first configuration (kappa = 0) the two stages err independently. In the second (kappa = 1.5) a hidden variable measuring how well the request is disguised lowers both scores together. Both stages use thresholds that block 1% of benign traffic, and a request is stopped if either stage flags it. The scores are synthetic and stand for no real classifier.

import numpy as np
rng = np.random.default_rng(3)
N, PREV = 200_000, 0.01
harm = rng.random(N) < PREV
hidden = rng.normal(size=N)                       # how well a harmful request is disguised

def scores(kappa):
    # prompt-stage and output-stage scores; kappa couples both stages to the same disguise
    sp = rng.normal(size=N) + harm * (2.5 - kappa * hidden)
    so = rng.normal(size=N) + harm * (2.5 - kappa * hidden)
    return sp, so

def run(kappa):
    sp, so = scores(kappa)
    tp = np.quantile(sp[~harm], 0.99)             # each stage blocks 1% of benign traffic
    to = np.quantile(so[~harm], 0.99)
    bp, bo = sp > tp, so > to
    miss = lambda b: float((~b[harm]).mean())
    fpr = float((bp | bo)[~harm].mean())
    print(f"kappa={kappa}: miss prompt-only {miss(bp):.3f}  output-only {miss(bo):.3f}  "
          f"both stages (block if either) {miss(bp | bo):.3f}  benign blocked {fpr:.3f}")
run(0.0)
run(1.5)

Program output (NumPy 2.5.3, Python 3.12; fixed seed):

kappa=0.0: miss prompt-only 0.422  output-only 0.414  both stages (block if either) 0.178  benign blocked 0.020
kappa=1.5: miss prompt-only 0.457  output-only 0.456  both stages (block if either) 0.329  benign blocked 0.020

With independent errors (kappa = 0) each stage misses about 42% of harmful requests, and the two together miss 17.8%, close to the product of the miss rates, 0.42×0.41 = 0.174. With a shared disguise (kappa = 1.5) each stage misses about 46% and the combination misses 32.9%, against 0.457² = 0.209 under independence. Benign traffic blocked is 2.0% in both cases, so the second stage costs a point of false positives for its gain. Real classifier errors correlate through the same mechanisms: prompts crafted to avoid a keyword filter are often also crafted to produce an output the visual classifier finds ambiguous. Test the stack as a whole against adversarial prompt sets, and report the miss rate of the combination, not the product of the individual miss rates. Threshold choices follow the usual trade-off between false positives and false negatives from Classification Metrics and Decision Thresholds, with the cost asymmetry set by the harm category: the acceptable miss rate for child safety content is not the one for a mildly off-style image.

Memorization, representation and misuse

Carlini et al. generate large numbers of images from public diffusion models and filter for near-duplicates of training images, and recover over a thousand training images from models including Stable Diffusion and Imagen. Duplicated training data raises the risk. Mitigations include deduplication before training and an output-time check that embeds each generated image and compares it with an index of protected images, using a similarity threshold tuned on held-out near-duplicates; embedding similarity explains the measure involved.

Representation is measured, not assumed. Generate from a fixed set of neutral prompts (for example, occupations) across several seeds, annotate the attributes of the people depicted, and compare the distributions with the intended target; the methodology is that of Fairness and Bias Auditing. Conditioning is part of the picture: a reference image or an identity adapter lets a user target a specific person, so those inputs deserve their own consent and abuse controls. Misuse that comes from outside the product, such as running an open-weight model locally, is out of the operator’s control, which is why provenance on the operator’s own outputs is useful but not a complete answer.

Release checks

The sections above reduce to a small set of gates that a release should pass. The table lists the gate, the measurement and the failure it is designed to catch.

Gate	Measurement	Failure it catches
Conditioning adherence	Pass rate per skill (count, color, position, binding) with prompt-bootstrap intervals; spatial-condition adherence via an estimator on the output	Fluent images that ignore the request; swapped attributes
Quality and coverage	FD or CMMD at a fixed sample count, with precision and recall, across a guidance sweep	Collapse under strong guidance; dropped modes
Temporal behavior	Adjacent-frame feature similarity, warping error, a motion statistic and per-frame quality, plus human review of identity and physics	Flicker, frozen clips, morphing identity, drift in long clips
Preference	Pairwise human study with intervals, per criterion	Metric gains that people do not perceive
Safety stack	Miss rate of the whole stack on adversarial prompt sets; false-positive rate on benign traffic; memorization screen	Correlated errors between stages; near-copies of training data
Provenance	Manifest validation after typical re-encodes; watermark detection rate and false-positive rate under realistic transformations	Marks lost by common platform processing; detectors without a measured error rate

References

• Zhang, Rao, and Agrawala (2023). Adding Conditional Control to Text-to-Image Diffusion Models (ControlNet).
• Ye, Zhang, Liu, Han, and Yang (2023). IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models.
• Ho, Salimans, Gritsenko, Chan, Norouzi, and Fleet (2022). Video Diffusion Models.
• Ho et al. (2022). Imagen Video: High Definition Video Generation with Diffusion Models.
• Heusel, Ramsauer, Unterthiner, Nessler, and Hochreiter (2017). GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium (introduces FID).
• Kynkäänniemi, Karras, Laine, Lehtinen, and Aila (2019). Improved Precision and Recall Metric for Assessing Generative Models.
• Jayasumana, Ramalingam, Veit, Glasner, Chakrabarti, and Kumar (2024). Rethinking FID: Towards a Better Evaluation Metric for Image Generation.
• Ge, Mahapatra, Parmar, Zhu, and Huang (2024). On the Content Bias in Fréchet Video Distance. CVPR.
• Huang et al. (2024). VBench: Comprehensive Benchmark Suite for Video Generative Models.
• Wen, Kirchenbauer, Geiping, and Goldstein (2023). Tree-Ring Watermarks: Fingerprints for Diffusion Images that are Invisible and Robust.
• Carlini et al. (2023). Extracting Training Data from Diffusion Models.
• Coalition for Content Provenance and Authenticity. C2PA Technical Specification, version 2.1.