Modality Gap

The modality gap is the observation that, in a trained contrastive model such as CLIP, image embeddings and text embeddings occupy two separate regions of the shared space, even though matched image-caption pairs are close to each other.