Word Embeddings

ByJu Yeon Eum

A one-hot vector identifies a vocabulary entry: its coordinate is 1 and every other coordinate is 0. Distinct one-hot vectors have cosine similarity 0 and Euclidean distance \(\sqrt2\), so that encoding alone expresses no graded word similarity. An embedding table learns a shorter vector for each entry. Looking up row 2 of a table with shape \((V,d)\) returns the \(d\)-dimensional vector for token ID 2; it is also the result of multiplying that token’s one-hot row by the table. Learned proximity can capture linguistic relationships, depending on the corpus and objective.

The distributional hypothesis

Words occurring in similar contexts often share semantic or grammatical properties. Context here means nearby tokens within a chosen window. Antonyms such as “hot” and “cold” can share contexts because they describe the same kind of property, so distributional proximity does not specifically mean synonymy. Predicting context encourages the vectors to encode patterns useful for that prediction.

These objectives obtain training targets from text without manual semantic labels. Predictive training also underlies language models, but static word-vector lookup and contextual sequence representations have different capacities.

How text becomes the integers a model actually sees is covered in Tokenization Explained: BPE, WordPiece, and SentencePiece.

Similarity and analogy

Cosine similarity compares direction and ignores magnitude:

\[\cos(u,v)=\frac{u^Tv}{\lVert u\rVert\lVert v\rVert}\]

For nonzero vectors, cosine removes the effect of length. Norm can reflect frequency, training choices, and other properties; it is not a universal frequency meter or necessarily irrelevant to a task. Raw dot product uses both norm and direction. Euclidean distance also uses both unless vectors are normalized, in which case squared distance equals 2 minus twice the cosine. Choose a comparison rule and validate it for the intended retrieval task.

import numpy as np

def cosine(u, v):
    u, v = np.asarray(u, dtype=float), np.asarray(v, dtype=float)
    if u.ndim != 1 or u.shape != v.shape:
        raise ValueError("expected matching vectors")
    if not (np.isfinite(u).all() and np.isfinite(v).all()):
        raise ValueError("vectors must be finite")
    nu, nv = np.linalg.norm(u), np.linalg.norm(v)
    if nu == 0 or nv == 0:
        raise ValueError("cosine is undefined for a zero vector")
    return float((u / nu) @ (v / nv))

def nearest(vec, embeddings, vocab, k=5, exclude=()):
    if k < 1:
        raise ValueError("k must be positive")
    scores = [(cosine(vec, embeddings[w]), w) for w in vocab if w not in exclude]
    return sorted(scores, key=lambda pair: (-pair[0], pair[1]))[:k]

def analogy(a, b, c, embeddings, vocab):
    """a is to b as c is to ?; raw-vector offset followed by cosine retrieval."""
    target = embeddings[b] - embeddings[a] + embeddings[c]
    return nearest(target, embeddings, vocab, k=1, exclude=(a, b, c))

print(round(cosine(np.array([1., 0.]), np.array([0.8, 0.6])), 3))
# 0.8

Analogy retrieval forms a query such as king minus man plus woman and searches for a nearby candidate. Excluding the input words changes the search space and is common in benchmark protocols. Report that rule, candidate vocabulary, relation categories, and whether source vectors were normalized before arithmetic. The helper above uses raw source vectors and cosine only at retrieval; other normalization choices can change the result. A correct answer on one analogy is not evidence that all relations are linear.

Word2Vec: skip-gram

Skip-gram predicts nearby context words from a center word. In “the cat sleeps”, a radius-1 window around “cat” gives the pairs (cat, the) and (cat, sleeps). CBOW combines surrounding-word representations to predict the center word. Window size, corpus size, token frequency, and training settings affect their relative performance; several pairs per occurrence do not guarantee good representations for rare words.

The naive objective uses a softmax over the entire vocabulary:

\[P(c\mid w)=\frac{e^{u_c^Tv_w}}{\sum_{j=1}^{V}e^{u_j^Tv_w}}\]

Here \(v_w\) is the center-word vector, \(u_c\) the context-word vector, and \(V\) the vocabulary size. There are two learned tables, each of shape \((V,d)\); input and output roles need not have the same vector. Exact softmax scores all context entries, costing \(O(Vd)\) per center representation with a direct implementation. Computation can be shared across its observed contexts, but the full-vocabulary normalization remains expensive for large \(V\).

Negative sampling

Skip-gram with negative sampling (SGNS) trains a binary discriminator between observed pairs and sampled noise pairs. It is a different objective from exact softmax, not an unbiased estimator of its loss. For an observed pair \((w,c)\), draw \(K\) context IDs from a noise distribution \(P_n\). Maximize the following expected score, or minimize its negative:

\[\log\sigma(u_c^Tv_w)+\sum_{k=1}^{K}\mathbb{E}_{n\sim P_n}\log\sigma(-u_n^Tv_w)\]

With sampled negatives the dot-product cost is \(O((K+1)d)\), excluding sampling and table storage. Noise words are training negatives, not verified semantic opposites: they can also be plausible contexts. The Word2Vec paper reported useful settings of 5–20 negatives for small datasets and 2–5 for large ones. Its noise distribution \(P_n(w)\propto f(w)^{3/4}\) smooths positive token counts \(f(w)\), increasing rare words’ relative probability compared with raw unigram sampling. The exponent is an empirical choice, not a universal optimum.

counts = np.array([1000.0, 100.0, 10.0, 1.0])
unigram = counts / counts.sum()
smoothed = counts ** 0.75 / (counts ** 0.75).sum()
print(np.round(unigram, 4))    
print(np.round(smoothed, 4))
# [9.001e-01 9.000e-02 9.000e-03 9.000e-04]
# [0.823  0.1464 0.026  0.0046]

Frequent-token subsampling removes some token occurrences before training-pair construction. It differs from smoothing the negative-sampling distribution: one changes observed training contexts, the other changes noise draws. Subsampling reduces repeated frequent-token work and can improve learned vectors; function words still carry grammatical information.

This toy batch contains two observed pairs and two noise draws per pair. Separate embedding tables supply center and context vectors. The code uses the negative of the score above, sums the negative terms within each pair, and averages over pairs. Sampling is with replacement and permits a noise ID to coincide with an observed context; implementations that reject collisions use a different sampling policy.

import torch
from torch import nn
import torch.nn.functional as F

torch.manual_seed(0)
centers = torch.tensor([0, 1])
contexts = torch.tensor([1, 2])
noise = torch.tensor(smoothed, dtype=torch.float32)
negatives = torch.multinomial(noise, 4, replacement=True).reshape(2, 2)
in_table, out_table = nn.Embedding(4, 3), nn.Embedding(4, 3)
optimizer = torch.optim.SGD(list(in_table.parameters()) + list(out_table.parameters()), lr=0.05)
v = in_table(centers)
pos = (v * out_table(contexts)).sum(-1)
neg = (v[:, None, :] * out_table(negatives)).sum(-1)
loss = -(F.logsigmoid(pos) + F.logsigmoid(-neg).sum(-1)).mean()
optimizer.zero_grad()
loss.backward()
print(tuple(pos.shape), tuple(neg.shape), bool(torch.isfinite(loss)))
# (2,) (2, 2) True
optimizer.step()

A high observed-pair score reduces its loss; a high noise-pair score increases its loss. At score zero, each binary term contributes log(2), so one positive and two negatives give about 2.079 per pair. This demonstrates one update, not learned word meaning. The sigmoid predicts observed-versus-noise membership under this sampling process, not the normalized probability of a context word. For downstream lookup, choose and document which trained table (or combination) is exported.

GloVe

GloVe builds a sparse global co-occurrence table by accumulating local context windows. An entry \(X_{ij}\) counts how often words \(i\) and \(j\) occur in those windows, possibly with distance weights. It learns two vector tables and biases by weighted least squares on positive counts:

\[J=\sum_{i,j}f(X_{ij})\left(u_i^Tv_j+b_i+\tilde b_j-\log X_{ij}\right)^2\]

With \(f(x)=\min(1,(x/x_{\max})^{3/4})\), rare observed pairs receive less weight, and the weight saturates at 1 for counts above \(x_{\max}\). Frequent pairs are not assigned decreasing weights. The sum excludes zero entries because \(\log0\) is undefined; the objective does not explicitly fit missing pairs to zero. For \(X_{ij}=4\), the target is \(\log4\approx1.386\). If the dot product plus biases is 1 and \(x_{\max}=100\), the weighted squared residual is about 0.01335. Sparsity helps, but constructing and storing co-occurrences can still be substantial work.

An idealized SGNS analysis lets each word–context dot product be optimized independently. For positive observed pair probability and positive noise probability, its finite optimum is \(\log[P(c\mid w)/(K P_n(c))]\). With unigram noise \(P_n(c)=P(c)\), this becomes pointwise mutual information (PMI), which compares the observed pair probability with what independence would predict, minus \(\log K\); the smoothed distribution changes that expression. Low-dimensional shared vectors constrain which scores can be realized, so training is not simply computing an exact matrix factorization. GloVe commonly exports the sum of its two word-role vectors, another choice to check when comparing pretrained files. FastText adds character n-gram representations, which can help with morphology and form vectors for unseen spellings using available subword features; it does not guarantee their semantic quality or outperform every word-level model.

Using pretrained embeddings

import torch
import torch.nn as nn

def build_embedding_layer(vocab, pretrained, dim, freeze=True):
    """Ordered unique tokens, with <pad> at 0 and <unk> at 1."""
    if dim < 1 or len(vocab) < 3 or list(vocab[:2]) != ["<pad>", "<unk>"]:
        raise ValueError("need positive dim, reserved tokens, and at least one word")
    if len(set(vocab)) != len(vocab):
        raise ValueError("vocab must contain unique tokens")
    rng = np.random.default_rng(0)
    matrix = rng.normal(0, 0.1, (len(vocab), dim)).astype(np.float32)
    matrix[0] = 0
    hits = 0
    for i, word in enumerate(vocab[2:], start=2):
        if word in pretrained:
            vector = np.asarray(pretrained[word], dtype=np.float32)
            if vector.shape != (dim,) or not np.isfinite(vector).all():
                raise ValueError(f"invalid vector for {word}")
            matrix[i] = vector
            hits += 1
    print(f"type coverage {hits}/{len(vocab)-2} = {hits/(len(vocab)-2):.1%}")
    return nn.Embedding.from_pretrained(torch.from_numpy(matrix), freeze=freeze, padding_idx=0)

vocab = ["<pad>", "<unk>", "cat", "kitten", "otter"]
pretrained = {"cat": np.array([1., 0.]), "kitten": np.array([0.8, 0.6])}
layer = build_embedding_layer(vocab, pretrained, dim=2)
# type coverage 2/3 = 66.7%
word_to_id = {w: i for i, w in enumerate(vocab)}
ids = torch.tensor([[word_to_id.get(w, 1) for w in ("cat", "unseen", "<pad>")]])
print(ids.tolist(), tuple(layer(ids).shape))
# [[2, 1, 0]] (1, 3, 2)
print(layer(ids)[0, 0].tolist(), layer(ids)[0, 2].tolist())
# [1.0, 0.0] [0.0, 0.0]

The example uses a tiny synthetic dictionary, not downloaded pretrained vectors. For a real file, load its word-to-vector mapping, preserve its dimension, and match casing and tokenization. Coverage here is the fraction of distinct ordinary vocabulary entries found, excluding reserved tokens. Also measure token-weighted coverage on the intended data: missing one frequent word can matter more than many rare entries. Low coverage can reflect domain or language mismatch as well as tokenization; there is no universal 80% diagnostic cutoff. Use an appropriate licensed source and document its corpus.

An in-vocabulary word missing from the pretrained file gets its own random row; a token outside this vocabulary maps to the shared unknown row. Freezing leaves both kinds of missing-word vectors untrained. Compare freezing, fine-tuning, or training missing rows on validation data. Fine-tuning may help even with a small dataset, though it can overfit or change useful relationships. Optimizer settings such as weight decay can also modify rows beyond those selected in a batch. The zero padding row does not remove padding from downstream pooling or loss calculations; mask those operations separately.

Bias

Embeddings reproduce the statistical associations of their training corpus, including harmful ones. Published analyses have found occupation terms aligned along a gender direction and name-based associations differing by ethnicity, in vectors trained on ordinary web text.

Corpus associations, preprocessing, and the learning objective all affect what an embedding captures. Matching corpus statistics is not evidence that using those associations in a downstream decision is appropriate.

Projecting out a selected bias direction can reduce a targeted association measure. Gonen and Goldberg showed that other recoverable structure can remain after such transformations. A lower intrinsic bias score therefore does not establish fairness of a downstream application. Document the corpus and mitigation method, evaluate relevant outcomes and uncertainty across affected groups, and assess remaining harms against the requirements of the intended use.

Distribution mismatch and transfer learning are treated separately in Data Mismatch, Transfer Learning, and Multitask Learning.

Static and contextual representations

A static word-level table assigns one vector to each vocabulary entry. The same “bank” vector is returned in river and finance contexts. It may encode aspects of multiple senses, but it is not necessarily their arithmetic average, and lookup alone cannot choose the intended sense for an occurrence. A sequence model consuming these vectors can still use surrounding words to disambiguate.

Contextual models produce occurrence-specific representations using surrounding tokens, so “bank” can receive different vectors in different sentences. Static tables remain useful for lightweight lookup or as inputs to a larger model. When choosing between them, consider whether the task needs an occurrence-specific representation and whether a downstream model already supplies that context.

Objectives that obtain supervision from data, including paired text–image supervision, are covered in Self-Supervised Learning: SimCLR, BYOL, MAE, and CLIP.

References

Discover more from Insightful Data Lab

Subscribe to get the latest posts sent to your email.

Similar Posts

Questions, corrections, or additional insights?

This site uses Akismet to reduce spam. Learn how your comment data is processed.