Modality Gap

The modality gap is the observation that, in a trained contrastive model such as CLIP, image embeddings and text embeddings occupy two separate regions of the shared space, even though matched image-caption pairs are close to each other. The model is trained to place a caption nearer to its own image than to other images, and that objective is satisfied while the two modalities stay at arm’s length from each other.

What is observed

In a contrastive model, an image encoder and a text encoder map their inputs to unit vectors, and training raises the cosine similarity of matched pairs relative to mismatched ones within a batch. Liang et al. (2022) report that the embeddings of the two modalities in such models form separate clusters: the mean image embedding and the mean text embedding are far apart, so that images sit near other images and captions near other captions. The distance between the two means (the centroid distance) is the usual summary of the gap.

Reported causes

Liang et al. attribute the gap to two factors.

1. Initialization. A randomly initialized deep network maps its inputs into a narrow cone of the output space, and the cone for the image encoder differs from the cone for the text encoder. The two modalities start with an offset.
2. Contrastive optimization. The training does not remove that offset, and the temperature parameter of the loss influences how large the gap stays; the authors show that changing the temperature changes the gap’s size.

A plausible reading is that the objective compares each caption with the other captions in the batch, and can favor relative separation without forcing the two modality means to coincide. Whether an offset changes the ranking depends on the candidate geometry.

How the gap affects ranking and similarity

A gap can coexist with good cross-modal retrieval, but a shared image offset does not guarantee unchanged caption rankings. For a query \(q\), offset \(a\), and caption \(d_j\), the dot product becomes \(q\cdot d_j+a\cdot d_j\); the added term generally differs across captions. Normalization can change the comparisons further. Rankings stay unchanged only under additional geometric conditions or when the perturbation is too small to cross the score gaps.

A hypothetical toy, run with numpy, shows this. Each of 500 concepts has a random 64-dimensional vector. The image embedding is the concept plus noise plus an offset \(+s\,u\), and the text embedding is the concept plus noise minus \(s\,u\), where \(u\) is a fixed unit direction, and all vectors are normalized. Increasing the offset scale \(s\):

import numpy as np
rng = np.random.default_rng(2)
n,d = 500,64
concept = rng.normal(size=(n,d))
x0 = concept + 0.3*rng.normal(size=(n,d))
y0 = concept + 0.3*rng.normal(size=(n,d))
u = rng.normal(size=d); u /= np.linalg.norm(u)
def norm(x): return x/np.linalg.norm(x,axis=1,keepdims=True)
def metrics(x,y):
    cross = x @ y.T; within = x @ x.T
    np.fill_diagonal(within,-np.inf)
    return (np.linalg.norm(x.mean(0)-y.mean(0)),np.diag(cross).mean(),
            np.mean(cross.argmax(1)==np.arange(n)),np.mean(within.max(1)>np.diag(cross)))
for s in (0,1,2,4):
    x,y = norm(x0+s*u),norm(y0-s*u)
    print(s,*(f"{v:.3f}" for v in metrics(x,y)))
x,y = norm(x-x.mean(0)),norm(y-y.mean(0))
print("centered",*(f"{v:.3f}" for v in metrics(x,y)))
q=np.array([1.,0.]); offset=np.array([0.,2.]); candidates=np.eye(2)
print("ranking counterexample",q@candidates.T,(q+offset)@candidates.T)

offset \(s\)	centroid distance	matched-pair cosine	image-to-text retrieval@1	images whose nearest other image beats their own caption
0	.018	.917	1.000	.000
1	.239	.889	1.000	.000
2	.466	.811	1.000	.000
4	.864	.554	1.000	.254

Retrieval stays perfect at the tested offsets for these generated vectors because each matched caption remains nearest. This is a result of the toy inputs, not a general invariance theorem. The final two-dimensional counterexample reverses a ranking from [1, 0] to [1, 2] after an offset. The gap shows up in other behavior: matched-pair similarity falls from 0.917 to 0.554, and at \(s=4\) for 25.4% of images the nearest other image is closer than the image’s own caption. Subtracting each modality’s mean embedding and renormalizing removes the gap in the toy (centroid distance 0.012) and retrieval stays at 1.000. The toy builds the offset in by hand; it does not show how a real model acquires a gap.

Effects

• Mixed-modality search. Searching a single index that holds both images and text ranks by cosine similarity. When same-modality items are closer than cross-modal ones, as in the separated clusters described above, results can be biased toward the query’s own modality.
• Embedding arithmetic. Averaging or adding an image embedding and a text embedding assumes both live in one comparable region; with a gap the result can lie off both clouds.
• Thresholds and calibration. Cross-modal cosine similarities can be lower than within-modality ones, so a similarity threshold tuned for one pairing may not transfer to the other.
• Downstream behavior. Liang et al. report that changing the gap distance affects zero-shot classification performance and fairness measurements, so the gap is not only a cosmetic property.

Remedies and cautions

Mean-centering each modality, as in the toy, is one candidate adjustment for same-index search, and it removes the offset in the toy. Whether closing the gap improves a given task depends on the model and data, so it has to be evaluated on that task. Closeness in a joint space is a property of the trained model and not a guarantee of semantic similarity, so operations that mix modalities in one space should be validated on the model in use.

References

• Liang, Zhang, Kwon, Yeung, Zou, “Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning” — https://arxiv.org/abs/2203.02053