Multimodal Models and Cross-Modal Alignment

Consider a hypothetical expense-reporting tool. A user photographs a receipt, speaks a question (“what was the total, and was tax included?”), and expects a typed answer. The language model at the center of the tool reads only one thing: a sequence of vectors, one per token. A photograph is a grid of pixel intensities. A voice recording is a long sequence of air-pressure samples. Neither is in the model’s vocabulary, and neither has the structure the model was trained on. Making this request answerable means turning pixels and samples into vectors that the language model can treat as if they were tokens, and doing so without destroying the information the answer depends on, such as the digits on the receipt.

That conversion has three parts. A modality encoder turns raw input into a sequence of feature vectors. An alignment mechanism maps those features into a space where the language model, or a text encoder, can use them. Input preparation decides what gets fed to the encoder in the first place: resolution, cropping, resampling, how many positions the result will occupy in the context window. After that come the questions of what the combined model can produce, how to measure whether it is using the image or audio at all, and where it fails. Each part has a design space, and each choice shows up later as a measurable behavior.

The article assumes the transformer block from Transformers from Scratch and the patch-token view of images from Vision Transformers and Modern Vision Models. It also assumes the experimental discipline of Fine-Tuning Experiments: Baselines, Ablations, and Failure Analysis: every claim that a multimodal model “understands” its image has to be earned by a comparison against a baseline that does not see the image. Here that discipline is applied to inputs rather than to adapter settings.

The shape of a multimodal model

Most current systems that accept images or audio and answer in text follow one pattern, shown below. A pretrained encoder for each modality produces features. A small trainable connector maps the features to the width of the language model’s embeddings. The projected vectors are placed in the input sequence in the positions of placeholder tokens, and the language model continues the sequence as usual.

raw image  --> resize / tile --> image encoder (ViT) --> patch features (N_v x D_v) --\
                                                                                      connector --> N x D_llm
raw audio  --> resample / log-mel --> audio encoder          --> frame features (N_a x D_a) --/        |
                                                                                                      v
text --> tokenizer --> token embeddings ------------------------> interleaved sequence --> language model --> text

Two other designs exist. In a joint embedding model, such as CLIP, the two encoders never feed a generator; they are trained so that matching image and text land near each other, which supports retrieval and classification but not free-form answers. In an any-to-any design, images or audio are converted to discrete tokens by a learned codec so that one autoregressive model can both read and write them. The connector pattern is the common starting point because it reuses two expensive pretrained components and trains only a small part, which is also why its failure modes are easy to localize: the encoder may not capture the detail, the connector may discard it, or the language model may ignore it.

Modality encoders

Images: from pixels to patch features

An image encoder reduces the sampling problem. A 224×224 RGB image has 50,176 positions; no one feeds that to a transformer directly. A vision transformer cuts the image into non-overlapping \(P\times P\) patches, projects each patch to a vector, and runs a transformer over the resulting sequence, as covered in the ViT article. For an image of height \(H\) and width \(W\), the encoder produces

\[ N_v = \frac{H}{P}\cdot\frac{W}{P} \]

feature vectors, each of width \(D_v\). With \(P=14\) and a 336×336 input, \(N_v = 24\times 24 = 576\). That number matters more than it first appears, because every one of those vectors can become a token in the language model’s context. The function below confirms the patch arithmetic and the ordering: patch 25 of the row-major sequence is the patch in row 1, column 1 of the grid, and its 588 values (\(14\times14\times3\)) are exactly that block of pixels.

Which encoder to use is mostly a question of what its pretraining taught it to represent. An encoder trained with a classification or contrastive objective emphasizes features that separate object categories and text-describable content; an encoder trained with a masked reconstruction objective keeps more local, low-level detail. Self-Supervised Learning: SimCLR, BYOL, MAE, and CLIP compares these objectives. Several widely used multimodal language models start from a contrastively pretrained image-text encoder such as CLIP’s ViT, because its features were already trained to be matched with language. LLaVA, for example, connects a CLIP ViT-L/14 encoder to a Vicuna language model (Liu et al.), and its 1.5 version keeps the encoder at 336-pixel input (Liu et al.).

Audio: from samples to frames

A speech or sound recording is a waveform sampled at a fixed rate, commonly 16,000 samples per second for speech. One second therefore holds 16,000 numbers, most of which are redundant for recognizing content: nearby samples differ by tiny amounts. The standard reduction is a time-frequency representation. Slice the waveform into overlapping windows, multiply each by a taper such as a Hann window, take a discrete Fourier transform to get energy per frequency, and group frequencies into bands spaced on the mel scale, which spaces bands more finely at low frequencies, where human hearing resolves pitch more precisely:

\[ m = 2595\,\log_{10}\!\left(1+\frac{f}{700}\right) \]

Here \(f\) is frequency in hertz and \(m\) is the mel value; the formula is a convention fitted to perceptual experiments, not a law. Taking the logarithm of band energies compresses the huge dynamic range of sound. The result is a log-mel spectrogram, a matrix with one row per window position and one column per mel band. With window length \(w\) samples and hop \(h\) samples, an input of \(n\) samples yields \(1+\lfloor (n-w)/h\rfloor\) frames.

Whisper, the speech recognizer described by Radford et al., is a concrete instance. It resamples audio to 16 kHz and computes an 80-channel log-magnitude mel spectrogram on 25-millisecond windows with a 10-millisecond stride. The encoder begins with a stem of two convolution layers of filter width 3, the second with stride two, followed by transformer blocks. The program below implements the front end from scratch with numpy on a synthetic signal that plays a 440 Hz tone for one second and then a 1,760 Hz tone for one second. The program’s purpose is to confirm the shapes and to show that the spectrogram really encodes frequency over time.

import numpy as np

def hz_to_mel(f):  return 2595.0 * np.log10(1.0 + f / 700.0)
def mel_to_hz(m):  return 700.0 * (10.0 ** (m / 2595.0) - 1.0)

def mel_filterbank(n_mels, n_fft, sr):
    mels = np.linspace(hz_to_mel(0), hz_to_mel(sr / 2), n_mels + 2)
    bins = np.floor((n_fft + 1) * mel_to_hz(mels) / sr).astype(int)
    fb = np.zeros((n_mels, n_fft // 2 + 1))
    for i in range(n_mels):
        l, c, r = bins[i], bins[i + 1], bins[i + 2]
        for k in range(l, c):
            fb[i, k] = (k - l) / max(c - l, 1)
        for k in range(c, r):
            fb[i, k] = (r - k) / max(r - c, 1)
    return fb

def log_mel(wave, sr=16000, win_ms=25, hop_ms=10, n_mels=80):
    win, hop = int(sr * win_ms / 1000), int(sr * hop_ms / 1000)   # 400, 160
    n_fft = 512
    n_frames = 1 + (len(wave) - win) // hop
    idx = np.arange(win)[None, :] + hop * np.arange(n_frames)[:, None]
    frames = wave[idx] * np.hanning(win)
    power = np.abs(np.fft.rfft(frames, n=n_fft)) ** 2             # (frames, 257)
    mel = power @ mel_filterbank(n_mels, n_fft, sr).T             # (frames, 80)
    return np.log10(np.maximum(mel, 1e-10))

sr = 16000
t = np.arange(2 * sr) / sr                       # 2 seconds
wave = np.where(t < 1.0, np.sin(2*np.pi*440*t), np.sin(2*np.pi*1760*t))  # 440 Hz then 1760 Hz
spec = log_mel(wave)
print("samples:", wave.shape[0], " log-mel shape (frames, mels):", spec.shape)
mel_centers = mel_to_hz(np.linspace(hz_to_mel(0), hz_to_mel(sr/2), 82))[1:-1]
for name, frame in [("t=0.5 s", 50), ("t=1.5 s", 150)]:
    b = spec[frame].argmax()
    print(f"{name}: strongest mel bin {b}, centre frequency about {mel_centers[b]:.0f} Hz")
# conv stem: second convolution has stride 2 -> frames halve
stem_frames = (spec.shape[0] + 1) // 2
print("frames after stride-2 stem:", stem_frames, "(each encoder step covers 20 ms)")
# a 30 s window at one frame per 10 ms (edge padding gives one frame per hop)
f30 = 30 * 1000 // 10
print("30 s window: log-mel frames", f30, "-> encoder positions", f30 // 2)

Program output (Python 3.12, numpy):

samples: 32000  log-mel shape (frames, mels): (198, 80)
t=0.5 s: strongest mel bin 15, centre frequency about 452 Hz
t=1.5 s: strongest mel bin 40, centre frequency about 1806 Hz
frames after stride-2 stem: 99 (each encoder step covers 20 ms)
30 s window: log-mel frames 3000 -> encoder positions 1500

The 2-second signal has 32,000 samples and becomes 198 frames of 80 values: the 100-frame-per-second rate follows from the 10-millisecond hop, and the two missing frames come from the window not fitting at the signal’s edges (production front ends pad the edges instead). The strongest mel band at 0.5 seconds has a center near 452 Hz, and at 1.5 seconds it has moved to a band near 1,806 Hz. Each band is wide, so the center frequency is only close to the tone, which is the point of a coarse representation. A stride-two convolution halves the frame rate, so each encoder step covers 20 milliseconds. A 30-second window therefore becomes 3,000 frames and 1,500 encoder positions, which is the sequence the encoder’s attention runs over.

The encoder output for audio is thus a sequence of feature vectors at tens of positions per second, in the same form as the patch features of an image. This regularity is why one connector design can serve both modalities: whatever the encoder, the language model sees a short sequence of vectors.

A second representation: discrete tokens

An encoder produces continuous features. A codec produces integers. A neural audio codec such as EnCodec (Défossez et al.) compresses audio with an encoder-decoder network whose latent space is quantized, so a recording becomes a sequence of codebook indices from which a decoder can reconstruct a waveform. The same idea applies to images with vector-quantized autoencoders. Discrete tokens matter for two reasons. They let a single next-token objective cover text, image and audio, and they make generation of non-text outputs a sampling problem over a vocabulary. They also cost sequence length. A hypothetical codec at 75 frames per second with 8 parallel codebooks emits 600 integers per second of audio; arranging or flattening them is a design decision in itself. Continuous features keep more information per position and need no codebook; discrete tokens trade some reconstruction fidelity for a uniform interface.

Alignment: making features usable by a language model

“Alignment” in this context means that vectors from different modalities live in a relationship where a downstream component can act on them: the text encoder can compare them with text, or the language model can read them as if they were tokens. There are two quite different goals behind the word, and the methods follow from which one you pick.

Joint embedding by contrastive training

The first goal is a shared space in which an image and its caption are close. CLIP trains an image encoder and a text encoder on image-caption pairs; the self-supervised article gives the training loop and the zero-shot classification recipe, and the embeddings article derives the contrastive objective for text retrieval. The core is a batch of \(N\) pairs. Let \(u_i\) and \(v_j\) be the unit-normalized image and text embeddings and \(s\) a scale equal to the inverse temperature. The score matrix is \(S_{ij}=s\,u_i^\top v_j\), and the loss is

\[ \mathcal{L} = \tfrac12\Big(\underbrace{-\tfrac1N\sum_i \log\frac{e^{S_{ii}}}{\sum_j e^{S_{ij}}}}_{\text{image}\to\text{text}} \;\underbrace{-\tfrac1N\sum_j \log\frac{e^{S_{jj}}}{\sum_i e^{S_{ij}}}}_{\text{text}\to\text{image}}\Big). \]

Each row is a classification problem whose correct class is the caption paired with that image; each column is the reverse. Every other item in the batch serves as a negative, so the loss depends on batch composition: with 64 items a model guessing uniformly has loss \(\ln 64\approx 4.16\). Zhai et al. propose a sigmoid alternative (SigLIP) that scores each image-text pair independently as a binary decision, which removes the normalization across the whole batch and so decouples the loss from batch size.

The program below trains this objective on synthetic paired data to make the mechanics visible. A latent vector \(z\) (8 dimensions) stands for the shared content. The “image feature” is \(zA\) and the “text feature” is \(zB\) for different random matrices, plus noise and a constant offset per modality. Two linear encoders map the 32-dimensional and 24-dimensional features into a 16-dimensional space. The gradients are derived by hand, including the unit-normalization step, and checked against finite differences before training.

import numpy as np
rng = np.random.default_rng(0)

# Synthetic paired data: a shared latent z generates an "image feature" x and a "text feature" y
# through different random linear maps, plus noise and a constant offset per modality.
d_z, d_x, d_y, d_e = 8, 32, 24, 16
A, B = rng.normal(size=(d_z, d_x)), rng.normal(size=(d_z, d_y))
mu_x, mu_y = 2.0 * rng.normal(size=d_x), 2.0 * rng.normal(size=d_y)
def make(n):
    z = rng.normal(size=(n, d_z))
    return z @ A + 0.5 * rng.normal(size=(n, d_x)) + mu_x, z @ B + 0.5 * rng.normal(size=(n, d_y)) + mu_y

def unit(E):
    n = np.linalg.norm(E, axis=1, keepdims=True)
    return E / n, n

def loss_and_grads(Wi, Wt, X, Y, scale):
    Ei, Et = X @ Wi, Y @ Wt
    U, ni = unit(Ei); V, nt = unit(Et)
    S = scale * U @ V.T
    N = len(X)
    def sm(M, ax):
        M = M - M.max(axis=ax, keepdims=True); P = np.exp(M); return P / P.sum(axis=ax, keepdims=True)
    Pr, Pc = sm(S, 1), sm(S, 0)                      # image->text rows, text->image columns
    idx = np.arange(N)
    loss = -0.5 * (np.log(Pr[idx, idx]).mean() + np.log(Pc[idx, idx]).mean())
    dS = 0.5 / N * ((Pr - np.eye(N)) + (Pc - np.eye(N)))
    dU, dV = scale * dS @ V, scale * dS.T @ U
    dEi = (dU - (dU * U).sum(1, keepdims=True) * U) / ni     # through the normalization
    dEt = (dV - (dV * V).sum(1, keepdims=True) * V) / nt
    return loss, X.T @ dEi, Y.T @ dEt

# gradient check against finite differences
Wi0, Wt0 = 0.1 * rng.normal(size=(d_x, d_e)), 0.1 * rng.normal(size=(d_y, d_e))
Xc, Yc = make(16)
l0, gi, gt = loss_and_grads(Wi0, Wt0, Xc, Yc, 14.0)
eps, errs = 1e-6, []
for _ in range(5):
    a, b = rng.integers(d_x), rng.integers(d_e)
    Wp = Wi0.copy(); Wp[a, b] += eps
    errs.append(abs((loss_and_grads(Wp, Wt0, Xc, Yc, 14.0)[0] - l0) / eps - gi[a, b]))
print("max |finite-difference - analytic| over 5 entries: %.2e" % max(errs))

def evaluate(Wi, Wt, n=500):
    X, Y = make(n)
    U, _ = unit(X @ Wi); V, _ = unit(Y @ Wt)
    S = U @ V.T
    r1 = (S.argmax(1) == np.arange(n)).mean()
    paired = np.diag(S).mean()
    unpaired = (S.sum() - np.trace(S)) / (n * n - n)
    gap = np.linalg.norm(U.mean(0) - V.mean(0))
    return r1, paired, unpaired, gap

def report(tag, Wi, Wt):
    r1, p, u, g = evaluate(Wi, Wt)
    print(f"{tag:>10}: recall@1 {r1:.3f} | mean cos paired {p:.3f} unpaired {u:.3f} | centroid distance {g:.3f}")

def train(scale, steps=1500):
    Wi, Wt = Wi0.copy(), Wt0.copy()
    m = [np.zeros_like(Wi), np.zeros_like(Wt)]; v = [np.zeros_like(Wi), np.zeros_like(Wt)]
    lr, b1, b2 = 0.01, 0.9, 0.999
    for step in range(1, steps + 1):
        X, Y = make(64)
        loss, gI, gT = loss_and_grads(Wi, Wt, X, Y, scale)
        for k, (W, g) in enumerate([(Wi, gI), (Wt, gT)]):
            m[k] = b1 * m[k] + (1 - b1) * g; v[k] = b2 * v[k] + (1 - b2) * g * g
            W -= lr * (m[k] / (1 - b1 ** step)) / (np.sqrt(v[k] / (1 - b2 ** step)) + 1e-8)
    return Wi, Wt

report("init", Wi0, Wt0)
for scale in (1.0, 1 / 0.07):
    Wi, Wt = train(scale)
    report(f"scale {scale:.1f}", Wi, Wt)
print("chance recall@1 with 500 candidates:", 1 / 500)

Program output (Python 3.12, numpy; synthetic data, fresh batches sampled at each step):

max |finite-difference - analytic| over 5 entries: 1.98e-06
      init: recall@1 0.000 | mean cos paired 0.041 unpaired -0.023 | centroid distance 0.757
 scale 1.0: recall@1 0.996 | mean cos paired 0.983 unpaired 0.005 | centroid distance 0.022
scale 14.3: recall@1 0.992 | mean cos paired 0.978 unpaired -0.000 | centroid distance 0.039
chance recall@1 with 500 candidates: 0.002

The gradient check agrees to about \(2\times10^{-6}\), consistent with the forward-difference step size, so the hand derivation is right. Before training, retrieval among 500 candidates is at zero: the encoders are random and the paired cosine similarity (0.041) is barely above the unpaired one (−0.023). After training, recall@1 is above 0.99, paired cosine is near 0.98 and unpaired near zero. The result holds for both a scale of 1 and a scale of 1/0.07 (temperature 0.07), because this task is easy: 8 latent dimensions generate everything both sides see.

The last column deserves a reading. Centroid distance measures how far the mean image embedding is from the mean text embedding on the unit sphere. At initialization it is 0.757 because the offsets \(\mu_x,\mu_y\) push the two modalities into different regions. Liang et al. report that in real models a corresponding modality gap persists after training, and attribute it to the narrow cone in which a randomly initialized deep network embeds its inputs together with the effect of the contrastive temperature. In this linear toy the gap closes (0.022 and 0.039), so a gap is not a necessary consequence of the loss. The toy shows that a small centroid gap can coexist with high retrieval accuracy; it does not establish invariance to moving the clouds. Adding an offset \(a\) to a query \(q\) changes each candidate score to \(q\cdot d_j+a\cdot d_j\). The added term can differ across candidates and change their order, and normalization can change it further. Rankings persist only when the geometry or score margins preserve that order. Embedding arithmetic between modalities and similarity thresholds also depend on their relative positions.

A joint-embedding model answers “which of these captions fits this image?” It does not generate an answer, and its single vector per image cannot say where in the image the evidence is. That limit motivates the second goal.

Connectors: giving a language model visual or acoustic tokens

The second goal is to let a pretrained language model condition on the encoder’s features. Three connector designs cover most systems.

Linear or MLP projection. Liu et al. (LLaVA) map each patch feature \(z\) to a language embedding with a trainable matrix, \(h = Wz\), keeping all patch positions. The follow-up LLaVA-1.5 replaces the single matrix with a small MLP, \(h = W_2\,\mathrm{GELU}(W_1 z)\). Every patch becomes a token, so a 336-pixel input with 14-pixel patches costs 576 positions in the context. Nothing is discarded by the connector; whatever the encoder encoded reaches the language model, and the language model’s own attention decides what to use.

A resampler with learned queries. Flamingo (Alayrac et al.) uses a Perceiver Resampler that turns a variable number of visual features into a fixed number of visual tokens, about 64 per image. BLIP-2 (Li et al.) uses a Querying Transformer, the Q-Former, with 32 learnable query vectors of width 768 that attend to the frozen encoder’s features through cross-attention. For a ViT-L/14, the encoder produces 257×1024 values, and the Q-Former output is 32×768, which the paper describes as an information bottleneck. A fully connected layer then projects the 32 vectors to the language model’s width, and they are prepended to the text embeddings as soft visual prompts. With queries \(Q\) and encoder features \(K=V\), each output vector is

\[ o_m = \sum_{i} \operatorname{softmax}_i\!\Big(\frac{q_m^\top k_i}{\sqrt{d}}\Big)\, v_i, \]

a weighted average over patches whose weights are chosen by a learned query. The number of output tokens is the number of queries, independent of image size.

Cross-attention layers inside the language model. Flamingo does not place visual tokens in the input sequence at all. It inserts new cross-attention layers between the frozen language-model blocks, so text positions attend to visual tokens at several depths, and uses a tanh gate on each new layer: \(y = x + \tanh(\alpha)\cdot\mathrm{CrossAttn}(x, V)\), where \(\alpha\) is a learned scalar that starts at zero so that, initially, the model computes exactly what the unmodified language model computes. Visual information then grows into the network during training. The cost of an image is paid in attention keys at the inserted layers rather than in context positions.

Connector	Tokens contributed per image	Trainable parameters	What it can lose	Example
Linear / MLP projection	All patches, e.g. 576 for a 336 px image at patch 14	A matrix or small MLP	Nothing at the connector; the language model must sift many tokens	LLaVA, LLaVA-1.5
Learned-query resampler	A fixed number, e.g. 32 or about 64	Queries plus cross-attention layers	Detail the queries do not learn to extract; counts and small text	BLIP-2, Flamingo (resampler)
Cross-attention in the language model	None in the input sequence	New attention layers per block, gated	Same as above, plus a more invasive change to the language model	Flamingo

The tension between the first two rows is a compression question, and a small experiment makes it concrete. The program below builds a 24×24 grid of random patch features (576 tokens) and adds a strong signature vector to exactly one patch, a stand-in for a small object. It then measures how much of that signature survives in the best output token under three schemes: 4×4 average pooling (36 tokens), a resampler with eight random queries, and a resampler in which one query was set by hand to point at the signature. The hand-built query is not learned; it illustrates what a trained query would have to do.

import numpy as np
rng = np.random.default_rng(1)
D, G = 64, 24                                   # feature width, 24x24 patch grid -> 576 tokens
feats = rng.normal(size=(G, G, D)) * 0.5        # background patch features
f = rng.normal(size=D); f *= 6.0 / np.linalg.norm(f)   # signature of a small object, norm 6
f_hat = f / np.linalg.norm(f)
feats[5, 17] += f                               # the object occupies exactly one patch
tokens = feats.reshape(-1, D)
print("visual tokens in:", tokens.shape[0])

def signal(out):
    """Largest projection of any output token onto the object direction, relative to |f|."""
    return (out @ f_hat).max() / np.linalg.norm(f)

# (a) 4x4 average pooling: 576 -> 36 tokens
pooled = feats.reshape(G // 4, 4, G // 4, 4, D).mean(axis=(1, 3)).reshape(-1, D)
print(f"avg-pool 4x4 : {pooled.shape[0]:3d} tokens, object signal kept {signal(pooled):.3f}")

# (b) cross-attention resampler with M queries: out_m = softmax(q_m K^T / sqrt(D)) V  (K = V = tokens here)
def resample(Q):
    logits = Q @ tokens.T / np.sqrt(D)
    w = np.exp(logits - logits.max(1, keepdims=True)); w /= w.sum(1, keepdims=True)
    return w @ tokens, w

Q_rand = rng.normal(size=(8, D))
out, _ = resample(Q_rand)
print(f"random queries:   8 tokens, object signal kept {signal(out):.3f}")

Q_match = Q_rand.copy(); Q_match[0] = 3.0 * f_hat * np.sqrt(D)   # one query that "looks for" the object
out, w = resample(Q_match)
print(f"one matched query: 8 tokens, object signal kept {signal(out):.3f}, "
      f"attention weight on the object patch {w[0, 5 * G + 17]:.3f}")

Program output (Python 3.12, numpy; synthetic features, seed 1):

visual tokens in: 576
avg-pool 4x4 :  36 tokens, object signal kept 0.100
random queries:   8 tokens, object signal kept 0.005
one matched query: 8 tokens, object signal kept 1.020, attention weight on the object patch 1.000

Average pooling over 16 patches divides the object’s contribution by 16, 0.0625 of its original strength before the background noise adds to the reading; the printed 0.100 includes that noise. Randomly directed queries keep almost nothing (0.005), because attention over 576 nearly uniform keys averages the object away. The matched query puts all its attention weight on the object patch and recovers the signature. Two conclusions follow from this hand-built example. First, compression by fixed pooling attenuates small, localized evidence in proportion to the pooled area, which is why tasks such as reading small text suffer when resolution or token count is cut. Second, a learned resampler can in principle keep small evidence, but only for the kinds of evidence its training signal rewarded; a query that was never useful for counting small objects will not have learned to find them.

How the pieces are trained

Connector designs come with a training schedule because the parts start in different conditions: a pretrained encoder, a pretrained language model, and a new connector with random weights. LLaVA uses two stages. In the first, both the encoder and the language model are frozen and only the projection matrix is trained on image-caption pairs converted to an instruction format (595,000 pairs filtered from CC3M); the authors describe this as training a compatible visual tokenizer for a frozen language model. In the second, the encoder stays frozen while the projection and the language model are updated on visual instruction data. BLIP-2 also uses two stages: the Q-Former is first trained against the frozen image encoder with image-text objectives, then connected to a frozen language model for generative training.

Three details of the loss matter for what the model learns. The language-model loss is computed on the response tokens only, not on the image slots or the prompt: \(\mathcal{L} = -\sum_{t\in\text{response}}\log p(x_t\mid \text{image tokens}, x_{<t})\). Image positions have no token target, so nothing directly forces the model to use them; it uses them only if they lower the loss on the response. If the answers in the training data can be predicted from the text alone, the model learns to ignore the image. Second, freezing the language model preserves its text abilities and cuts trainable parameters, while unfreezing it, whether fully or through low-rank adapters as in Fine-Tuning and Alignment: SFT, LoRA, RLHF, and DPO, allows the model to adapt to instruction formats and to new visual vocabulary at the cost of possible regression on text tasks. Third, the evidence for which stage helps with what comes from ablations: freeze or unfreeze each part, change the connector, change the data mix, and compare to a baseline. The experimental design in the fine-tuning experiments article applies unchanged, with the added requirement that each ablation is also run on the control conditions described later.

Input preparation

The encoder sees whatever the preprocessing hands it, and the language model sees only a number of positions that preprocessing decided. Input preparation is therefore part of the model’s behavior, not plumbing. Four decisions carry most of the consequences.

Resolution, aspect ratio, and tiling

A fixed-size encoder forces a choice for non-square or large images. Resizing the whole image to a square distorts aspect ratio and shrinks fine detail. Padding preserves proportions but wastes positions on blank area. Tiling splits the image into several crops at the encoder’s native resolution, optionally adds one downscaled copy of the whole image for global context, and feeds all of them, so the token count grows with the number of tiles. The function below implements one hypothetical tiling policy: choose a grid of tiles whose aspect ratio is closest to the image’s, never use more tiles than the native resolution would need, and add a global thumbnail when there is more than one tile. Real systems vary in the details; the point is the token arithmetic.

import math
import numpy as np

def patchify(img, p):
    """(H, W, C) -> (num_patches, p*p*C), row-major order."""
    H, W, C = img.shape
    assert H % p == 0 and W % p == 0
    x = img.reshape(H // p, p, W // p, p, C).transpose(0, 2, 1, 3, 4)
    return x.reshape(-1, p * p * C)

img = np.random.default_rng(0).random((336, 336, 3))
patches = patchify(img, 14)
print("patches:", patches.shape)                       # (576, 588)
# patch 25 is row 1, column 1 of the 24x24 grid
assert np.array_equal(patches[25], img[14:28, 14:28].reshape(-1))

def visual_tokens(h, w, patch=14, tile=336, max_tiles=6, thumbnail=True):
    """Tokens when the image is split into tile x tile crops (grid chosen to match aspect ratio)."""
    best = None
    cap = min(max_tiles, math.ceil(h / tile) * math.ceil(w / tile))   # no more tiles than native pixels need
    for r in range(1, cap + 1):
        for c in range(1, cap // r + 1):
            err = abs(math.log((w / h) / (c / r)))
            if best is None or err < best[0] or (err == best[0] and r * c > best[1] * best[2]):
                best = (err, r, c)
    _, r, c = best
    per_tile = (tile // patch) ** 2
    n_tiles = r * c + (1 if thumbnail and r * c > 1 else 0)
    return r, c, n_tiles * per_tile

print("single 336x336 resize:", (336 // 14) ** 2, "tokens")
for h, w in [(336, 336), (1080, 1920), (3000, 800), (2000, 2000)]:
    r, c, n = visual_tokens(h, w)
    print(f"{w}x{h} image -> {r} rows x {c} cols of tiles (+thumbnail), {n} visual tokens")

# Interleaved sequence assembly with a context budget
IMG, AUD = "<image>", "<audio>"
def assemble(parts, budget, n_img=576, n_aud=750):
    seq, kinds = [], []
    for kind, content in parts:
        if kind == "text":
            toks = content.split()
        elif kind == "image":
            toks = [IMG] * n_img
        else:
            toks = [AUD] * n_aud
        seq += toks; kinds += [kind] * len(toks)
    if len(seq) > budget:
        raise ValueError(f"sequence {len(seq)} tokens exceeds budget {budget}")
    return seq, kinds

parts = [("text", "Here is the receipt"), ("image", None),
         ("text", "and my spoken question"), ("audio", None), ("text", "What is the total ?")]
seq, kinds = assemble(parts, budget=4096)
print("total tokens:", len(seq), "| text:", kinds.count("text"),
      "| image:", kinds.count("image"), "| audio:", kinds.count("audio"))
# where do the placeholders sit? their embeddings get replaced by projected encoder outputs
pos = [i for i, k in enumerate(kinds) if k == "image"]
print("image slots occupy positions", pos[0], "to", pos[-1])
try:
    assemble(parts * 4, budget=4096)
except ValueError as e:
    print("error:", e)

Program output (Python 3.12, numpy):

patches: (576, 588)
single 336x336 resize: 576 tokens
336x336 image -> 1 rows x 1 cols of tiles (+thumbnail), 576 visual tokens
1920x1080 image -> 1 rows x 2 cols of tiles (+thumbnail), 1728 visual tokens
800x3000 image -> 4 rows x 1 cols of tiles (+thumbnail), 2880 visual tokens
2000x2000 image -> 2 rows x 2 cols of tiles (+thumbnail), 2880 visual tokens
total tokens: 1339 | text: 13 | image: 576 | audio: 750
image slots occupy positions 4 to 579
error: sequence 5356 tokens exceeds budget 4096

A 336×336 input is 576 tokens. A 1920×1080 screenshot becomes two tiles plus a thumbnail, 1,728 tokens, and a tall 800×3000 image becomes four tiles plus a thumbnail, 2,880. The assembly function after the tiling code shows the second half of the problem. A receipt photo of 576 tokens plus a spoken question of 750 audio tokens (a hypothetical 15 seconds at 50 positions per second) plus 13 text tokens is 1,339 positions. Four such turns exceed a 4,096-token budget and the function raises, which is the behavior a serving system needs to have deliberately: when media tokens crowd out the conversation, something has to be dropped, downscaled or refused, and the choice should not be an accident of whichever call hits the limit first.

Token count is also a cost model. Attention scores scale with the square of the sequence length, so a prompt of 200 text tokens with one 576-token image has about \(776^2\approx 6.0\times10^5\) pairwise scores per head and layer, and with a 2,880-token tiled image \(3080^2\approx 9.5\times10^6\), about 15.8 times as many. The cost of long inputs and the way long contexts degrade are covered in Context Windows, Position, and Long-Context Behavior.

Normalization and the training-serving match

Encoders are trained with a specific preprocessing pipeline: a resampling filter, a channel order, and per-channel mean and standard deviation normalization for images; a sample rate, window, hop, mel-band count, and global scaling for audio. Changing any of them at inference changes the numbers the encoder sees without raising an error. A 44.1 kHz recording fed to a front end that assumes 16 kHz does not fail; it looks to the encoder like audio slowed down by a factor of about 2.8 and lowered in pitch, and recognition quality drops quietly. The same applies to a different image resampling filter, or to RGB versus BGR. This is a form of training-serving skew, and the remedy is the same: take the preprocessing from the model’s own specification, share one implementation between training and serving, and test it with fixed inputs whose expected feature values are known.

Duration, padding, and windows for audio

A transformer over audio frames has a cost that grows with the square of the number of frames. Whisper handles long recordings by working in 30-second windows and padding or truncating each window to that length. A system that accepts an hour of audio must therefore chunk it, and chunk boundaries cut words in half and split the context a model needs to resolve ambiguity. Overlapping chunks and merging the results is the usual remedy, at the price of more compute and a merge step that can duplicate or drop words at seams.

Sequence assembly

Once the encoders and connectors have produced vectors, they must be put into the sequence. The usual mechanism uses placeholder tokens such as <image> in the tokenized prompt. The model looks up an embedding for each placeholder as it would for any token, and then the embedding is replaced by the corresponding projected feature vector. The tokenizer, the placeholder count and the projected feature count must agree exactly; a mismatch is a classic source of silent errors. Position handling follows the language model: image tokens occupy consecutive positions like text, so their order within the image grid is whatever order the patches were flattened in. For tiled images the order of tiles is part of the interface the model was trained with. A training-time attention mask can also stop later images from seeing earlier ones, as in Flamingo’s per-image masking, and the same mask must be applied at inference.

Generation

In the connector pattern, the output is text. The language model decodes a response token by token conditioned on the image, audio and prompt, with the sampling controls from standard language-model decoding. This covers captioning, question answering, document reading, transcription and spoken-content summarization. The visual or acoustic input changes the conditional distribution, but the decoder is the same.

Producing an image or audio as output needs a different mechanism, and there are two common ones. One keeps the single autoregressive model and extends its vocabulary with codec or image-tokenizer indices, so that the model emits discrete image or audio tokens and a separate decoder converts them back to pixels or waveform. The sequence-length cost from the codec discussion applies here: generating 10 seconds from the hypothetical 600-tokens-per-second codec takes 6,000 sequential tokens. The other attaches a separate generator, typically a diffusion or flow model, to the language model: the language model produces a text or embedding that conditions the generator. Diffusion Models Explained covers the sampler and guidance machinery these generators use, including latent diffusion, where a decoder maps generated latents back to pixels. Conditioning, temporal consistency and quality measures for those generators are a separate subject; the evaluation concern here is the same in both designs: whether the condition actually influences the output.

Evaluating a multimodal model

A single accuracy number on a visual question-answering benchmark does not tell you whether the model reads the image. A model can do well because the questions are predictable from text alone. Goyal et al. showed this for VQA by building a balanced dataset in which each question has two similar images with different answers, and found that models that did well on the original benchmark did significantly worse on the balanced one. The evaluation design below makes the same check routine.

Metrics by task

Task	Common metric	What it measures	What it hides
Image-text retrieval	Recall@K: fraction of queries whose correct item is in the top K	Ranking in a joint space	Sensitivity to the candidate pool; no localization
Visual question answering	Exact-match or normalized accuracy	Short-answer correctness	Reliance on language priors; answer-format mismatch
Captioning	n-gram overlap with reference captions	Similarity to human references	Correct but differently phrased captions; hallucinated details that happen to overlap
Object hallucination	Yes/no polling accuracy, precision, recall, yes-rate (POPE)	Tendency to assert objects that are absent	Hallucination of attributes and relations
Speech recognition	Word error rate	Edit distance between hypothesis and reference	Errors that change meaning versus ones that do not
Generated media	Distribution-level and condition-adherence scores	Quality and fit to the prompt	Failures on rare conditions; memorization

Recall@K and classification-style metrics are covered in the embeddings article and Classification Metrics and Decision Thresholds. POPE, introduced by Li et al., polls a model with multiple yes/no questions of the form “is there an X in the image?”, including questions about objects that are absent, and so isolates object hallucination from generation style.

Controls that test whether the input is used

The idea is to rerun every evaluation with the visual input degraded in ways that should destroy the answer. Three conditions are enough. In the real condition the question is paired with its image. In the blank condition the image is replaced by a uniform one. In the shuffled condition the image is replaced by that of another random example. A model that reads the image should drop to a prior-only level in the last two. A model that stays near its real-image score is not using the image for that question type, and the gap between real and shuffled is an estimate of how much the image contributes. The same idea applies to audio: replace the clip with silence or with another clip.

The program below runs this harness on a synthetic benchmark of 900 questions with a deliberately skewed answer distribution: existence questions are balanced, count answers cluster at 2, and the color “red” is the most common. There is no real model. Two stub models stand in: model A consults the image on 95 percent of questions and model B on 40 percent, falling back otherwise to a fixed language prior (always “yes”, “2”, “red”); when they do read a real image they are correct with a per-type probability. The numbers therefore illustrate how the harness works, not how any real system performs.

import random
from collections import Counter

rnd = random.Random(0)
COLORS = ["red", "blue", "green", "yellow", "white", "black"]
# Synthetic benchmark: 900 questions about 900 images. The label distribution is skewed on purpose
# (as it often is in real data), so a model that never looks at the image can still score well.
def make_item(i):
    qtype = ["exists", "count", "color"][i % 3]
    if qtype == "exists":
        ans = "yes" if rnd.random() < 0.5 else "no"            # balanced, POPE-style
    elif qtype == "count":
        ans = str(rnd.choices(range(6), weights=[5, 20, 40, 20, 10, 5])[0])
    else:
        ans = rnd.choices(COLORS, weights=[35, 20, 15, 10, 12, 8])[0]
    return {"id": i, "qtype": qtype, "answer": ans}
data = [make_item(i) for i in range(900)]

PRIOR = {"exists": "yes", "count": "2", "color": "red"}      # the language prior of the stub models
def wrong_answer(item, r):
    pool = {"exists": ["yes", "no"], "count": [str(k) for k in range(6)], "color": COLORS}[item["qtype"]]
    return r.choice([a for a in pool if a != item["answer"]])

def make_stub(p_use_image, p_correct_given_image):
    """Stub model. With probability p_use_image it 'reads' the image and is right with
    p_correct_given_image; otherwise it answers from the language prior. Image absent or from another
    example: the evidence it reads is wrong, so it can only be right by chance or by the prior."""
    def model(item, image_condition, r):
        if r.random() < p_use_image:
            if image_condition == "real":
                return item["answer"] if r.random() < p_correct_given_image[item["qtype"]] else wrong_answer(item, r)
            if image_condition == "shuffled":      # evidence from a different example
                other = data[r.randrange(len(data))]
                return other["answer"] if other["qtype"] == item["qtype"] else PRIOR[item["qtype"]]
            return PRIOR[item["qtype"]]            # blank image carries no evidence
        return PRIOR[item["qtype"]]
    return model

def run(model, condition, seed=1):
    r = random.Random(seed)
    return [(it, model(it, condition, r)) for it in data]

def accuracy(rows, qtype=None):
    rows = [(it, p) for it, p in rows if qtype in (None, it["qtype"])]
    return sum(p == it["answer"] for it, p in rows) / len(rows)

def exists_stats(rows):
    ex = [(it, p) for it, p in rows if it["qtype"] == "exists"]
    tp = sum(p == "yes" and it["answer"] == "yes" for it, p in ex)
    fp = sum(p == "yes" and it["answer"] == "no" for it, p in ex)
    fn = sum(p == "no" and it["answer"] == "yes" for it, p in ex)
    return tp / (tp + fp), tp / (tp + fn), sum(p == "yes" for _, p in ex) / len(ex)

pc = {"exists": 0.92, "count": 0.55, "color": 0.88}
models = {"A (uses image 95%)": make_stub(0.95, pc), "B (uses image 40%)": make_stub(0.40, pc)}

print("answer distribution (counts):", dict(Counter(it['answer'] for it in data if it['qtype'] == 'count')))
print("prior-only baseline (always yes / 2 / red):",
      f"{sum(PRIOR[it['qtype']] == it['answer'] for it in data) / len(data):.3f}")
print()
for name, m in models.items():
    print(name)
    for cond in ("real", "shuffled", "blank"):
        rows = run(m, cond)
        print(f"  {cond:>8}: overall {accuracy(rows):.3f} | exists {accuracy(rows,'exists'):.3f}"
              f" count {accuracy(rows,'count'):.3f} color {accuracy(rows,'color'):.3f}")
    prec, rec, yes_rate = exists_stats(run(m, "real"))
    print(f"  exists, real image: precision {prec:.3f} recall {rec:.3f} fraction answered yes {yes_rate:.3f}")

# Uncertainty: bootstrap over questions for the A-B difference on real images
rowsA, rowsB = run(models["A (uses image 95%)"], "real"), run(models["B (uses image 40%)"], "real")
cA = [p == it["answer"] for it, p in rowsA]; cB = [p == it["answer"] for it, p in rowsB]
rb, diffs = random.Random(7), []
for _ in range(2000):
    idx = [rb.randrange(len(data)) for _ in data]
    diffs.append(sum(cA[i] for i in idx) / len(idx) - sum(cB[i] for i in idx) / len(idx))
diffs.sort()
print(f"A minus B, real images: {sum(cA)/len(cA) - sum(cB)/len(cB):.3f}, 95% bootstrap interval "
      f"[{diffs[49]:.3f}, {diffs[1949]:.3f}]")

Program output (Python 3.12, standard library only; stub models on synthetic data):

answer distribution (counts): {'3': 55, '2': 109, '4': 37, '1': 60, '0': 26, '5': 13}
prior-only baseline (always yes / 2 / red): 0.394

A (uses image 95%)
      real: overall 0.774 | exists 0.920 count 0.527 color 0.877
  shuffled: overall 0.364 | exists 0.510 count 0.320 color 0.263
     blank: overall 0.394 | exists 0.517 count 0.363 color 0.303
  exists, real image: precision 0.917 recall 0.929 fraction answered yes 0.523
B (uses image 40%)
      real: overall 0.560 | exists 0.647 count 0.467 color 0.567
  shuffled: overall 0.390 | exists 0.517 count 0.353 color 0.300
     blank: overall 0.394 | exists 0.517 count 0.363 color 0.303
  exists, real image: precision 0.600 recall 0.948 fraction answered yes 0.817
A minus B, real images: 0.214, 95% bootstrap interval [0.174, 0.256]

Start with the baseline. A model that never sees an image and answers “yes”, “2”, “red” scores 0.394 overall on this benchmark, and about 0.36 on count questions (the share of “2” answers) even though no counting happens. Any real-image score has to be read against that floor, not against chance. Model A scores 0.774 with real images and 0.364 with shuffled ones, close to the floor, so its gain comes from the image. Model B scores 0.560 with real images and 0.390 shuffled; it uses the image on a minority of questions, and its existence questions show the failure pattern known from hallucination work: it answers “yes” to 0.817 of existence questions although half are false, with precision 0.600 against recall 0.948. Overall accuracy would have ranked B below A anyway, but the yes-rate is what tells you the kind of error, and so what to fix.

The bootstrap line quantifies the A-versus-B difference over resampled questions: 0.214 with a 95 percent interval of 0.174 to 0.256. The interval reflects question sampling noise for this fixed set of stub outputs. It does not capture variation from retraining, which requires repeated runs with different seeds. The same breakdown by question type, the same controls, and the same interval reasoning are what the experiment-analysis article asks of any adapter comparison.

Data problems specific to multimodal evaluation

The dataset concerns from Designing Fine-Tuning Datasets and Evaluation Splits carry over and gain a few forms. The same image can appear in training and test splits under different questions, so splits should be by image, or by source document or recording, not by question. Benchmarks built from public web images may overlap with the web-scale caption data used to pretrain the encoder. Near-duplicate images, re-encoded or cropped, defeat exact-match deduplication, so duplicate detection should operate on embeddings of the media rather than on file hashes. Finally, human-written reference captions describe what people notice, which is not the same as what is in the image, so a metric based on overlap with references measures agreement with that habit.

Limitations

Each limitation below follows from a mechanism already described, which is why it can be diagnosed rather than only observed.

• Information bottleneck. The encoder’s resolution, the pooling or query count in the connector, and the context budget decide the finest detail that can reach the language model. Small text, thin lines, dense charts and distant objects fail first. Raising resolution grows the token count with the square of the image side, and attention cost grows with the square of the token count; tiling grows tokens linearly with the number of tiles. The pooling experiment above is the mechanism.
• Language priors. The training loss is on response tokens, so any regularity in the text can substitute for looking. Models answer the common answer. The blank and shuffled controls catch this; the balanced-pair design of Goyal et al. prevents it in the benchmark itself.
• Hallucination. A model asserts objects, attributes or text that are not present, in proportion to how plausible they are given the rest of the scene. POPE-style polling measures object hallucination, with the yes-rate and precision to separate bias from error. The fix has to match the cause: a connector bottleneck, a prior-dominated training set and a decoding setting call for different changes.
• Spatial relations and counting. Patch features carry position only through the positional encodings of the encoder, and a token sequence flattens the grid. Left-of, in-front-of and exact counts are more fragile than recognizing which objects are present. These are the question types where the stub experiment’s count accuracy is lowest by construction, and real benchmarks should be broken down the same way.
• Alignment is only as good as its data. Web captions are noisy, short, and often describe context rather than content. A connector trained on them learns what captions mention. Contrastive training also treats other captions in a batch as negatives even when they describe the same scene, which caps what the embedding can distinguish.
• Geometry of the embedding space. The modality gap reported by Liang et al. is a reminder that closeness in a joint space is a property of the trained model, not of semantics. Operations that mix modalities in one space, such as averaging an image embedding with a text embedding, should be validated for the model in use.
• Conflicting evidence. When the text prompt and the image disagree, a model has no principled rule for which wins. Text embedded in an image can carry instructions the user did not write, so an image is an untrusted input in the same way a retrieved document is.
• Privacy and rights. Photographs reveal faces, locations and metadata; voice recordings identify speakers and can be cloned. Training data for these modalities carries consent and license questions that the dataset-rights discussion in the dataset article applies to as well.
• Cost and latency. Media tokens consume context and compute, so latency and price rise with image count, resolution and audio duration. The token arithmetic from the input section is the first estimate to make before choosing a design.

These points also define what a first experiment should look like. Fix the encoder and language model, train only the connector on a small, clean caption set, and measure on a benchmark broken down by question type with blank and shuffled controls. Then change one thing at a time: connector type, token count, resolution, which parts are frozen. Each step has a measurable question and a baseline to beat.

References

• Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision (CLIP). ICML. arXiv:2103.00020.
• Zhai, Mustafa, Kolesnikov, and Beyer (2023). Sigmoid Loss for Language Image Pre-Training (SigLIP). ICCV. arXiv:2303.15343.
• Liu, Li, Wu, and Lee (2023). Visual Instruction Tuning (LLaVA). NeurIPS. arXiv:2304.08485.
• Liu, Li, Li, and Lee (2023). Improved Baselines with Visual Instruction Tuning (LLaVA-1.5). arXiv:2310.03744.
• Li, Li, Savarese, and Hoi (2023). BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. ICML. arXiv:2301.12597.
• Alayrac et al. (2022). Flamingo: a Visual Language Model for Few-Shot Learning. NeurIPS. arXiv:2204.14198.
• Radford et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision (Whisper). arXiv:2212.04356.
• Défossez, Copet, Synnaeve, and Adi (2022). High Fidelity Neural Audio Compression (EnCodec). arXiv:2210.13438.
• Liang, Zhang, Kwon, Yeung, and Zou (2022). Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning. NeurIPS. arXiv:2203.02053.
• Li et al. (2023). Evaluating Object Hallucination in Large Vision-Language Models (POPE). EMNLP. arXiv:2305.10355.
• Goyal, Khot, Summers-Stay, Batra, and Parikh (2017). Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. CVPR. arXiv:1612.00837.