GloVe: Learning Word Embeddings from Global Co-Occurrence Statistics

GloVe, short for Global Vectors for Word Representation, learns word embeddings from aggregated word co-occurrence statistics.

Where Skip-Gram learns from sampled center–context pairs, GloVe first summarizes the corpus in a co-occurrence matrix. It then learns vectors whose dot products approximate the logarithms of the observed co-occurrence counts.

From Local Windows to Global Statistics

Consider a corpus containing tokens\[ w_1,w_2,\ldots,w_T. \]

For every token occurrence, examine nearby words within a context window. These local observations are accumulated over the entire corpus.

Define\[ X_{ij} \]

as the weighted number of times word \(j\) appears in the context of word \(i\).

The resulting matrix\[ X\in\mathbb{R}^{V\times V} \]

is called the co-occurrence matrix, where \(V\) is the vocabulary size.

A large \(X_{ij}\) means that words \(i\) and \(j\) frequently occur near each other.

Building the Co-Occurrence Matrix

Suppose the corpus contains:

“I want a glass of orange juice.”

With a context-window radius of two, the word “orange” may co-occur with:

• “glass”
• “of”
• “juice”

Each observed pair increases the corresponding matrix entry.

A simple unweighted update is\[ X_{ij}\leftarrow X_{ij}+1. \]

However, closer words are often given more weight. If words \(i\) and \(j\) are \(r\) positions apart, one possible update is\[ X_{ij} \leftarrow X_{ij}+\frac{1}{r}. \]

This weighting gives immediate neighbors more influence than distant words within the window.

Symmetric and Asymmetric Contexts

Whether the co-occurrence matrix is symmetric depends on how context is defined.

Symmetric window

If context includes words on both sides, then a pair may be counted in both directions:\[ X_{ij}\approx X_{ji}. \]

With fully symmetric processing, equality may hold exactly.

Directional context

If context includes only preceding words, then\[ X_{ij} \]

and\[ X_{ji} \]

can differ.

Directional co-occurrence can preserve information about word order, while symmetric windows emphasize general proximity.

Why Co-Occurrence Is Informative

Words with related meanings often have similar co-occurrence profiles.

For example, “ice” and “steam” may both co-occur with “water,” but their relationships with “solid” and “gas” differ.

The ratios of co-occurrence probabilities can reveal these distinctions:\[ \frac{P(k\mid i)}{P(k\mid j)}. \]

If context word \(k\) is strongly associated with \(i\) but not \(j\), this ratio becomes large. If it is strongly associated with \(j\) but not \(i\), the ratio becomes small. If it is similarly associated with both, the ratio approaches 1.

This co-occurrence-ratio perspective helped motivate GloVe’s use of vector differences and dot products.

The Basic GloVe Relationship

GloVe aims to learn vectors so that their dot products reflect log co-occurrence counts.

Let:

• \(w_i\) be the word vector for word \(i\)
• \(\tilde{w}_j\) be the context vector for word \(j\)
• \(b_i\) be the word bias
• \(\tilde{b}_j\) be the context bias

The target relationship is\[ w_i^\top\tilde{w}_j + b_i + \tilde{b}_j \approx \log X_{ij}. \]

The bias terms are part of the standard GloVe formulation. They help absorb effects related to the overall frequencies of words and contexts.

Why Use the Logarithm?

Co-occurrence counts have a highly skewed distribution.

A common word pair may appear millions of times, while a rare but meaningful pair may appear only a few times. Directly fitting raw counts would allow extremely frequent pairs to dominate.

The logarithm compresses this range:\[ \log(1)=0, \]\[ \log(100)\approx4.61, \]\[ \log(1{,}000{,}000)\approx13.82. \]

Although one million is 10,000 times larger than 100, their logarithms are much closer.

This makes the regression problem more manageable and reflects relative differences more naturally.

The GloVe Objective

The complete objective is\[ J = \sum_{i=1}^{V} \sum_{j=1}^{V} f(X_{ij}) \left( w_i^\top\tilde{w}_j + b_i + \tilde{b}_j – \log X_{ij} \right)^2. \]

The parameters are:\[ w_i,\quad \tilde{w}_j,\quad b_i,\quad \tilde{b}_j. \]

The model minimizes a weighted least-squares error between:

• The learned score
• The logarithm of the observed co-occurrence count

Handling Zero Counts

If\[ X_{ij}=0, \]

then\[ \log X_{ij} \]

is undefined.

GloVe avoids evaluating those entries by assigning them zero weight:\[ f(0)=0. \]

Operationally, training iterates only over nonzero entries of the sparse co-occurrence matrix.

This is important because most possible word pairs never occur together. Explicitly processing every zero would be computationally wasteful.

The Weighting Function

Not every observed count should contribute equally.

The standard GloVe weighting function is\[ f(x) = \begin{cases} \left(\dfrac{x}{x_{\max}}\right)^\alpha, & x<x_{\max},\\[8pt] 1, & x\geq x_{\max}. \end{cases} \]

Typical historical choices include\[ \alpha=\frac{3}{4} \]

and a selected cutoff \(x_{\max}\).

The function has several useful properties:

• \(f(0)=0\)
• Rare pairs receive nonzero but reduced weight
• Weight increases smoothly with count
• Very frequent pairs are capped at weight 1
• Extremely common words cannot dominate without limit

The constants are hyperparameters rather than universal laws.

Why Rare Pairs Receive Less Weight

A pair observed once may represent:

• A meaningful rare relationship
• A spelling error
• A tokenization artifact
• An accidental juxtaposition
• Noise in the corpus

Giving every rare event the same weight as a frequently confirmed pair would make the model sensitive to noise.

However, ignoring all rare pairs would lose valuable information. The weighting function provides a compromise.

Why Frequent Pairs Are Capped

Common function words such as “the,” “of,” and “is” produce many co-occurrences.

Without a cap, these high-frequency relationships could dominate the squared-error objective. GloVe allows their evidence to matter while preventing their counts from overwhelming the optimization.

Word and Context Vectors

GloVe learns two vectors for every vocabulary item:\[ w_i \]

and\[ \tilde{w}_i. \]

The first represents the token in the word role, while the second represents it in the context role.

This resembles the two embedding tables used by Skip-Gram:

• Input embeddings
• Output embeddings

If the context definition is symmetric, the two roles are structurally similar, although the learned vectors need not be identical.

Constructing the Final Embedding

After training, possible final representations include:\[ e_i=w_i, \]\[ e_i=\tilde{w}_i, \]

or\[ e_i=w_i+\tilde{w}_i. \]

An average can also be used:\[ e_i = \frac{w_i+\tilde{w}_i}{2}. \]

The sum and average differ only by a global scaling factor, which disappears under cosine similarity.

Combining both vectors often uses information learned in both roles.

Symmetry of the Objective

When the co-occurrence matrix and context definition are symmetric, the word and context roles can be exchanged:\[ w_i \leftrightarrow \tilde{w}_i, \]\[ b_i \leftrightarrow \tilde{b}_i. \]

The objective retains essentially the same structure.

This symmetry motivates combining the two learned representations after optimization.

However, the two tables should not be assumed to converge to identical values. Their relationship depends on initialization, optimization, context construction, and the invariances of the factorization.

GloVe as Weighted Matrix Factorization

The model can be viewed as approximately factorizing the log co-occurrence matrix.

Ignoring biases and weights for intuition,\[ w_i^\top\tilde{w}_j \approx \log X_{ij}. \]

If all word vectors are arranged into a matrix \(W\) and all context vectors into \(\tilde{W}\), then\[ W\tilde{W}^\top \approx \log X. \]

The actual objective is more refined because it includes:

• Bias terms
• A weighting function
• Only nonzero co-occurrences
• Low-dimensional factors

This connects GloVe to classical matrix-factorization methods while retaining a prediction-inspired embedding geometry.

Why the Objective Can Learn Meaning

At first glance, fitting log counts with squared error may appear too simple to capture semantic structure.

The key is that each vector participates in many equations.

For word \(i\), the model simultaneously tries to satisfy relationships involving all contexts \(j\) for which\[ X_{ij}>0. \]

Two words with similar co-occurrence profiles face similar constraints. Their learned vectors therefore tend to occupy related regions.

The structure comes from the full system of co-occurrence relationships, not from any single matrix entry.

Comparing GloVe and Skip-Gram

Both methods learn from nearby word relationships, but they organize the learning problem differently.

The distinction should not be exaggerated. Both ultimately exploit local co-occurrence patterns aggregated across a corpus.

GloVe makes the global count structure explicit, while Skip-Gram learns from sampled local events.

Global Does Not Mean Whole-Sentence Understanding

The word “global” in GloVe refers to aggregating statistics across the full corpus.

It does not mean that each vector understands an entire document or models arbitrary long-range dependencies. The underlying counts are still generated from local context windows.

A large corpus-wide co-occurrence matrix summarizes many local observations.

Efficient Sparse Training

A vocabulary of size \(V\) yields a conceptual matrix with\[ V^2 \]

possible entries. For\[ V=100{,}000, \]

that would be\[ 10^{10} \]

entries.

Most are zero, so a dense matrix is impractical. Implementations store only nonzero entries:\[ (i,j,X_{ij}). \]

Training iterates over these sparse triples.

The main pipeline is:

1. Tokenize the corpus.
2. Build vocabulary counts.
3. Generate weighted co-occurrence records.
4. Aggregate duplicate pairs.
5. Store nonzero entries sparsely.
6. Optimize the weighted regression objective.
7. Combine word and context vectors.

Gradient Structure

For one nonzero pair, define the residual\[ r_{ij} = w_i^\top\tilde{w}_j + b_i + \tilde{b}_j – \log X_{ij}. \]

The pairwise loss is\[ J_{ij} = f(X_{ij})r_{ij}^2. \]

Its gradients are\[ \frac{\partial J_{ij}}{\partial w_i} = 2f(X_{ij})r_{ij}\tilde{w}_j, \]\[ \frac{\partial J_{ij}}{\partial \tilde{w}_j} = 2f(X_{ij})r_{ij}w_i, \]\[ \frac{\partial J_{ij}}{\partial b_i} = 2f(X_{ij})r_{ij}, \]\[ \frac{\partial J_{ij}}{\partial \tilde{b}_j} = 2f(X_{ij})r_{ij}. \]

Only the parameters associated with that observed pair need to be updated.

A Simplified Pairwise Loss

import torch

def glove_pair_loss(
    word_vectors,
    context_vectors,
    word_biases,
    context_biases,
    counts,
    x_max=100.0,
    alpha=0.75
):
    weights = torch.clamp(
        (counts / x_max) ** alpha,
        max=1.0
    )

    scores = (
        (word_vectors * context_vectors).sum(dim=1)
        + word_biases
        + context_biases
    )

    residuals = scores - torch.log(counts)

    return (
        weights * residuals.square()
    ).mean()

import torch

def glove_pair_loss(
    word_vectors,
    context_vectors,
    word_biases,
    context_biases,
    counts,
    x_max=100.0,
    alpha=0.75
):
    weights = torch.clamp(
        (counts / x_max) ** alpha,
        max=1.0
    )

    scores = (
        (word_vectors * context_vectors).sum(dim=1)
        + word_biases
        + context_biases
    )

    residuals = scores - torch.log(counts)

    return (
        weights * residuals.square()
    ).mean()

This function assumes all counts are strictly positive. Zero-count pairs should not be included.

Why Individual Dimensions Are Not Interpretable

A simplified explanation of embeddings may assign labels such as:

• Gender
• Royalty
• Food
• Animacy

to separate coordinates. Real learned dimensions generally do not align with these concepts.

Suppose the dot-product model uses vectors \(w_i\) and \(\tilde{w}_j\). For any invertible matrix \(A\), define\[ w_i’=Aw_i \]

and\[ \tilde{w}_j’ = A^{-\top}\tilde{w}_j. \]

Then\[ (w_i’)^\top\tilde{w}_j’ = (Aw_i)^\top (A^{-\top}\tilde{w}_j). \]

Expanding gives\[ w_i^\top A^\top A^{-\top}\tilde{w}_j = w_i^\top\tilde{w}_j. \]

Thus, the dot products remain unchanged.

The objective cannot uniquely determine one human-interpretable coordinate system. Features may be rotated, scaled, or mixed across dimensions while preserving the fitted pair scores.

A Nuance About Combining the Two Tables

The inverse-transformation argument preserves cross-table products between \(w_i\) and \(\tilde{w}_j\). It does not necessarily preserve the geometry of\[ w_i+\tilde{w}_i \]

under arbitrary invertible transformations.

This reinforces the point that the learned parameterization is not fully identifiable. Interpretability and downstream geometry depend partly on optimization conventions and the way the final vectors are constructed.

For transformations applied identically and orthogonally to the final embeddings, cosine similarities and analogy structure are preserved.

Why Analogies Can Still Appear

Although individual coordinates are difficult to interpret, meaningful relationships can be distributed across the space.

For example,\[ e_{\text{woman}}-e_{\text{man}} \approx e_{\text{queen}}-e_{\text{king}}. \]

This relationship does not require one coordinate to mean “gender.” It requires only that the relevant difference be encoded consistently as a direction in the vector space.

Distributed representations can therefore support useful similarity and analogy behavior without providing axis-by-axis explanations.

Hyperparameters That Matter

Important GloVe settings include:

• Vocabulary size
• Minimum token frequency
• Context-window radius
• Symmetric or directional context
• Distance weighting
• Embedding dimension
• \(x_{\max}\)
• Weighting exponent \(\alpha\)
• Learning rate
• Number of training passes
• Tokenization method

These choices influence whether embeddings emphasize local syntax, broader semantics, rare words, or high-frequency structure.

Advantages of GloVe

GloVe offers several appealing properties:

• A conceptually simple objective
• Explicit use of corpus-wide co-occurrence counts
• Efficient training over sparse nonzero entries
• Useful semantic and syntactic geometry
• Clear connection to matrix factorization
• Reusable pretrained static embeddings

Limitations of GloVe

GloVe also has important limitations:

• Each vocabulary item receives one static vector.
• Polysemous words combine multiple meanings.
• Out-of-vocabulary words require special handling.
• Building the co-occurrence structure can require substantial memory.
• Results depend heavily on tokenization and corpus quality.
• Embeddings can encode harmful social biases.
• Local windows do not capture every long-range relationship.
• The model does not directly produce contextual token representations.

Subword and contextual models address several of these limitations.

Common Mistakes

Omitting the bias terms

The standard objective includes both word and context biases:\[ b_i+\tilde{b}_j. \]

Taking the logarithm of zero

Only positive co-occurrence entries should be processed.

Building a dense \(V\times V\) matrix

The co-occurrence structure should normally be stored sparsely.

Assuming the matrix must be symmetric

Symmetry depends on how context is defined.

Interpreting every coordinate as a named feature

The embedding basis is not uniquely determined.

Assuming “global” means full-document context

GloVe aggregates local-window counts over the corpus.

Allowing frequent pairs to dominate

The weighting function is central to balancing frequent and rare observations.

Key Takeaway

GloVe learns word and context vectors by fitting weighted log co-occurrence counts:\[ J = \sum_{i,j} f(X_{ij}) \left( w_i^\top\tilde{w}_j + b_i + \tilde{b}_j – \log X_{ij} \right)^2. \]

The co-occurrence matrix summarizes how often words appear near one another across the corpus. The logarithm compresses the large count range, while the weighting function balances rare and frequent pairs.

The learned dimensions are not individually interpretable because many coordinate transformations preserve the model’s pairwise scores. Nevertheless, relationships can emerge in the overall geometry, producing useful similarities and approximate analogy directions.