InfoNCE Loss

InfoNCE is the contrastive loss behind most embedding models and CLIP-style image-text models. It turns “make matching pairs close and non-matching pairs far” into a classification problem: given a query, pick the one correct item out of a set of candidates containing one positive and several negatives. Van den Oord, Li, and Vinyals introduced it in contrastive predictive coding, where it trains a representation to predict future parts of a signal while negative sampling keeps the computation tractable.

The formula

Let a query \(q\) have one positive \(d^+\) and \(K\) negatives \(d_1^-,\dots,d_K^-\). Let \(s(\cdot,\cdot)\) be a similarity, usually a dot product or cosine of the two embeddings, and \(\tau>0\) a temperature. The loss is

\[ \mathcal{L} = -\log \frac{\exp(s(q,d^+)/\tau)}{\exp(s(q,d^+)/\tau)+\sum_{k=1}^{K}\exp(s(q,d_k^-)/\tau)}. \]

This is softmax cross-entropy over \(K+1\) candidates in which the correct class is the positive. The loss is small when the positive’s similarity is much larger than every negative’s, and it is large when some negative scores close to or above the positive.

In a batch of \(B\) pairs \((q_i,d_i)\), every other pair’s document serves as a negative for \(q_i\) (“in-batch negatives”), so \(K=B-1\) with no extra encoding. Models often add the symmetric loss with the roles of queries and documents swapped, as CLIP does for images and texts.

Three things to read off the formula

Temperature. Dividing similarities by \(\tau\) before the softmax sharpens or flattens the distribution. The program below uses one query with hypothetical cosine similarities 0.70 to the positive and 0.60, 0.20, 0.10 to three negatives.

import math
def softmax(z):
    m=max(z); e=[math.exp(x-m) for x in z]; s=sum(e); return [x/s for x in e]
s=[0.70,0.60,0.20,0.10]
for tau in (1.0,0.2,0.05):
    p=softmax([x/tau for x in s]); print(f"tau={tau:<5} probs={[round(x,3) for x in p]} loss={-math.log(p[0]):.3f}")

Output:

tau=1.0   probs=[0.327, 0.296, 0.198, 0.179] loss=1.118
tau=0.2   probs=[0.575, 0.349, 0.047, 0.029] loss=0.553
tau=0.05  probs=[0.881, 0.119, 0.0, 0.0] loss=0.127

At \(\tau=1\) the softmax is almost flat, so the loss barely distinguishes the near-miss negative (0.60) from the clearly wrong ones. At \(\tau=0.05\) the near-miss takes essentially all the remaining probability and the easy negatives vanish. The gradient with respect to the logit of candidate \(j\) is \(P_j-\mathbf{1}[j=+]\): each negative is pushed away in proportion to the probability the model currently assigns it. A small temperature concentrates learning on the hardest negatives, and also makes the loss punish a mis-ranked pair harder. Too small a temperature makes training unstable and amplifies the effect of mislabeled negatives; many models learn the temperature or fix it between 0.01 and 0.1.

Batch size. With more negatives the task is harder. The next program fixes the positive at cosine 0.7, every negative at 0.2, \(\tau=0.1\), and varies the number of candidates \(B\).

for B in (2,8,64,512):
    z=[0.7/0.1]+[0.2/0.1]*(B-1); print(f"B={B:4d} loss={-math.log(softmax(z)[0]):.4f}  log B={math.log(B):.3f}")

B=   2 loss=0.0067  log B=0.693
B=   8 loss=0.0461  log B=2.079
B=  64 loss=0.3538  log B=4.159
B= 512 loss=1.4914  log B=6.238

The same scores yield a larger loss as the candidate set grows. When the scores carry no information, all \(B\) candidates are equally likely and the loss is \(\log B\) (4.1589 for \(B=64\), the chance level). Van den Oord et al. derive a mutual-information lower bound \(I(q;d) \ge \log B-\mathbb{E}[\mathcal{L}]\) when the positive comes from the joint query-document distribution and negatives are independent draws from the document marginal. More negatives raise the bound’s ceiling; a single batch loss is not itself a population guarantee. Hard-negative mining changes the sampling distribution, so that bound cannot be assumed unchanged. Large batches or memory banks still provide more comparisons for representation learning.

False negatives. The loss assumes every other candidate is a true negative. If a batch contains another document that is actually relevant to \(q\), the loss pushes the model to score it below the labeled positive, which damages the representation. Deduplicating near-identical pairs and checking mined hard negatives reduces this.

Relation to nearby losses

• It is ordinary softmax cross-entropy, with candidates formed by similarity scores instead of a fixed set of classes. The “classes” change with every query.
• A triplet or margin loss compares one negative at a time and ignores how far it is beyond the margin. InfoNCE weighs all negatives together and smoothly.
• It is a ranking-flavored surrogate, not a retrieval metric. Doing well at picking the positive among a batch tends to carry over to ranking in a large corpus, but nothing guarantees the metric value.

References

• van den Oord, A., Li, Y., Vinyals, O., Representation Learning with Contrastive Predictive Coding — https://arxiv.org/abs/1807.03748