Word Representations and the Foundations of Word Embeddings

Natural language models cannot operate directly on words as written text. Words must first be converted into numerical representations that a learning algorithm can process.

A simple approach is to represent each word with a one-hot vector. Although this uniquely identifies every vocabulary item, it does not capture relationships between words. Word embeddings address this limitation by mapping words into a dense vector space where related words can receive related representations.

Representing Words with a Vocabulary

Suppose a model uses a vocabulary containing \(V\) tokens:\[ \mathcal{V} = \{w_1,w_2,\ldots,w_V\}. \]

Each token is assigned an integer index. For example:

The precise indices are arbitrary. They identify vocabulary entries but do not express anything about meaning.

One-Hot Representations

If “man” has index 5,391, its one-hot representation is\[ o_{\text{man}} = o_{5391} \in \mathbb{R}^{V}. \]

This vector contains a 1 at position 5,391 and zeros everywhere else.

In general,\[ (o_i)_j = \begin{cases} 1, & i=j,\\ 0, & i\neq j. \end{cases} \]

One-hot vectors have several useful properties:

• Every vocabulary item has a unique representation.
• Token lookup is conceptually simple.
• The representation contains no accidental ordering.
• Sparse operations can be efficient.

However, their main weakness is serious:

One-hot vectors encode token identity but do not encode semantic or syntactic similarity.

Why One-Hot Vectors Do Not Capture Similarity

Consider the words:

• apple
• orange
• king
• queen

Intuitively, “apple” should be more closely related to “orange” than to “king.” One-hot vectors cannot express this.

For two different vocabulary words \(i\neq j\),\[ o_i^\top o_j=0. \]

Thus,\[ o_{\text{apple}}^\top o_{\text{orange}}=0, \]

but also\[ o_{\text{apple}}^\top o_{\text{king}}=0. \]

Every distinct pair has the same dot product.

Their Euclidean distance is also identical:\[ \lVert o_i-o_j\rVert_2 = \sqrt{2}, \qquad i\neq j. \]

Therefore,\[ \lVert o_{\text{apple}}-o_{\text{orange}} \rVert_2 = \lVert o_{\text{apple}}-o_{\text{king}} \rVert_2. \]

From the geometry of the one-hot representation, all different words are equally unrelated.

Why This Limits Generalization

Suppose a language model has frequently observed:

“I want a glass of orange juice.”

The model may learn that “juice” often follows “orange.” Now consider:

“I want a glass of apple …”

Humans can immediately transfer what they know because apples and oranges share many properties:

• Both are fruits.
• Both can be squeezed or processed.
• Both commonly occur with the word “juice.”
• Both appear in similar grammatical contexts.

A one-hot representation does not expose these similarities. The model must learn separate relationships for “orange juice” and “apple juice,” unless its architecture and data provide another way to connect them.

A better representation should make it easier to reuse patterns across related words.

Featurized Word Representations

Instead of representing a word as a sparse vocabulary identifier, suppose each word is represented by a dense vector of learned features:\[ e_w\in\mathbb{R}^{d}. \]

For illustration, imagine dimensions loosely corresponding to properties such as:

• Gender association
• Royalty
• Age association
• Food
• Animacy
• Size
• Grammatical role

A hypothetical representation might look like this:

This table is only an intuition-building example. Real embedding dimensions are learned automatically and normally do not have clean labels such as “royalty” or “food.”

A learned word vector might contain hundreds of values:\[ e_{\text{apple}} = \begin{pmatrix} 0.18\\ -0.42\\ 0.73\\ \vdots\\ 0.09 \end{pmatrix} \in\mathbb{R}^{300}. \]

The useful information is distributed across many coordinates.

What Is a Word Embedding?

A word embedding maps a vocabulary item into a point in a continuous vector space:\[ w\longmapsto e_w\in\mathbb{R}^{d}. \]

The word “embedding” reflects the idea that discrete objects are placed, or embedded, in a continuous geometric space.

For example:\[ \text{apple} \longmapsto e_{\text{apple}}, \]\[ \text{orange} \longmapsto e_{\text{orange}}. \]

If the training objective is effective, related words often occupy nearby or structurally related regions.

The model can then exploit relationships such as\[ e_{\text{apple}} \approx e_{\text{orange}}, \]

while maintaining greater separation from an unrelated representation:\[ e_{\text{apple}} \not\approx e_{\text{king}}. \]

The Embedding Matrix

For a vocabulary of size \(V\) and embedding dimension \(d\), store all embeddings in a matrix:\[ E\in\mathbb{R}^{V\times d}. \]

The row associated with token \(i\) is its embedding:\[ e_i=E[i,:]. \]

An alternative mathematical convention stores embeddings as columns:\[ E\in\mathbb{R}^{d\times V}, \qquad e_i=Eo_i. \]

Both conventions describe the same idea. Software libraries commonly store one embedding per row.

If\[ V=50{,}000 \]

and\[ d=300, \]

then\[ E\in\mathbb{R}^{50{,}000\times300}. \]

The model learns 300 values for each vocabulary item.

Embedding Lookup

Although multiplication by a one-hot vector selects an embedding mathematically, practical systems do not construct the one-hot vector.

Instead, the model performs a direct lookup:

import torch.nn as nn

embedding = nn.Embedding(
    num_embeddings=50_000,
    embedding_dim=300
)

vectors = embedding(token_ids)

import torch.nn as nn

embedding = nn.Embedding(
    num_embeddings=50_000,
    embedding_dim=300
)

vectors = embedding(token_ids)

If token_ids has shape\[ (B,T), \]

the embedding output has shape\[ (B,T,300). \]

Here:

• \(B\) is the batch size.
• \(T\) is the stored sequence length.
• 300 is the embedding dimension.

Dense Does Not Always Mean Cheaper

A 300-dimensional embedding is much smaller than a 50,000-dimensional one-hot vector. However, one-hot vectors are extremely sparse and are usually represented implicitly as indices.

The main benefit of embeddings is therefore not simply computational compression.

The fundamental advantage is that embeddings contain learned relational structure.

A dense vector may actually require more storage per token occurrence than an integer token ID. The gain comes from the reusable embedding table and the meaningful geometry it provides to later layers.

Learning Similarity from Context

Word embeddings are commonly motivated by the distributional hypothesis:

Words used in similar contexts tend to have related meanings or functions.

Consider these sentence patterns:

• “She ate an apple.”
• “She ate an orange.”
• “The apple was sweet.”
• “The orange was sweet.”
• “Fresh apples are sold here.”
• “Fresh oranges are sold here.”

Because “apple” and “orange” repeatedly occur in similar contexts, an embedding algorithm can learn related vectors for them.

However, contextual similarity is not identical to synonymy. Words can be close because they:

• Have similar meanings
• Belong to the same category
• Serve similar grammatical roles
• Frequently co-occur
• Appear in parallel expressions
• Are opposites used in similar contexts

For example, “hot” and “cold” may receive similar vectors because both commonly describe temperature.

Measuring Similarity

A common similarity measure is cosine similarity:\[ \operatorname{sim}(u,v) = \frac{u^\top v} {\lVert u\rVert_2\lVert v\rVert_2}. \]

Cosine similarity compares vector directions rather than raw magnitudes.

Its range is\[ -1\leq\operatorname{sim}(u,v)\leq1. \]

Broadly:

• A value near 1 indicates similar directions.
• A value near 0 indicates little directional similarity.
• A value near -1 indicates opposite directions.

Euclidean distance can also be used:\[ d(u,v)=\lVert u-v\rVert_2. \]

The most appropriate measure depends on how the embeddings were trained and normalized.

Embeddings Enable Statistical Generalization

Suppose a model learns a function\[ f(e_{w_1},e_{w_2},\ldots,e_{w_T}). \]

If related words have similar embeddings, replacing one word with a related word produces a nearby input representation.

For example,\[ e_{\text{apple}} \approx e_{\text{orange}} \]

means that a pattern learned for “orange juice” may help with “apple juice.”

This is not a logical guarantee. The model still needs suitable training data and architecture. But the representation gives it a much better starting point than unrelated one-hot identifiers.

Embeddings Can Encode Multiple Relationships

The geometry of an embedding space can capture several kinds of structure simultaneously:

• Semantic category
• Grammatical behavior
• Typical usage
• Domain association
• Morphological similarity
• Social and cultural associations

For example, the representations of “king” and “queen” may be related because both refer to royalty, while also differing along patterns associated with gender.

These relationships are distributed across the vector space. Individual coordinates rarely correspond cleanly to a single human-interpretable property.

Word Analogies

Certain embedding spaces exhibit approximate vector relationships such as\[ e_{\text{king}}-e_{\text{man}} + e_{\text{woman}} \approx e_{\text{queen}}. \]

This suggests that some semantic relationships can appear as directions in the vector space.

However, analogy behavior should not be overstated:

• It depends on the training method and corpus.
• It is sensitive to the similarity metric.
• It does not demonstrate general logical reasoning.
• Some famous examples are unusually favorable.
• The same space can encode undesirable stereotypes.

Analogies are best viewed as evidence that embeddings can capture regular geometric relationships, not as proof that every linguistic concept is organized through simple vector arithmetic.

Visualizing High-Dimensional Embeddings

An embedding with 100, 300, or more dimensions cannot be directly plotted. A dimensionality-reduction method can map the vectors into two dimensions for visualization.

One widely used method is t-SNE. It attempts to preserve local neighborhoods, so words that are close in the original space may appear near one another in a two-dimensional plot.

A visualization may show clusters of:

• Fruits
• Animals
• Countries
• Occupations
• Numbers
• Related verbs
• Names

This can provide an intuitive view of local embedding structure.

Interpreting t-SNE Carefully

A t-SNE plot can be useful, but its geometry must not be interpreted too literally.

Important limitations include:

• Distances between widely separated clusters may not be meaningful.
• Cluster sizes can be visually misleading.
• Results can change with initialization and hyperparameters.
• Two-dimensional projection discards substantial information.
• Apparent clusters do not necessarily correspond to clean semantic categories.

t-SNE is primarily a tool for exploring local neighborhoods, not a faithful map of the complete high-dimensional space.

Nearest-neighbor calculations and quantitative evaluations should be performed in the original embedding space rather than on the two-dimensional projection.

Static Word Embeddings

Traditional word embeddings associate each vocabulary entry with one fixed vector:\[ w\longmapsto e_w. \]

This means a word such as “bank” receives the same representation in:

• “She visited the bank to withdraw money.”
• “They sat beside the river bank.”

The fixed-vector approach cannot directly represent the different meanings.

Static embeddings remain useful because they are:

• Compact
• Fast
• Easy to inspect
• Effective with smaller models
• Suitable for limited-compute environments

Their context independence is nevertheless an important limitation.

Contextual Representations

A contextual model computes a token representation from its surrounding sequence:\[ e_t = f(w_t,w_1,\ldots,w_T). \]

The representation of “bank” can therefore change according to whether the surrounding words concern finance or a river.

Contextual embeddings capture:

• Word meaning in context
• Syntactic role
• Nearby entity information
• Sentence-level relationships
• Position-specific usage

Modern NLP systems frequently use contextual representations rather than one fixed vector per word.

Word, Subword, and Character Units

The discrete units being embedded do not have to be complete words.

Word embeddings

Each vocabulary word has its own entry. This is simple but handles rare and unseen words poorly.

Subword embeddings

Words are divided into reusable pieces. This makes it possible to represent uncommon words without assigning every complete word a separate entry.

Character embeddings

Individual characters are embedded and composed into larger representations. This can capture spelling and morphology but produces longer sequences.

Subword tokenization is a widely used compromise between vocabulary size and sequence length.

Unknown Words

A fixed word vocabulary traditionally maps unseen words to an unknown token:\[ \text{unseen word}\longrightarrow\text{<UNK>}. \]

This causes many unrelated words to share the same representation.

Possible improvements include:

• Increasing the vocabulary
• Using subword tokenization
• Constructing embeddings from characters
• Applying morphology-aware models
• Using contextual encoders

Names, technical terms, and newly created words particularly benefit from subword or character-level processing.

Embeddings Can Encode Bias

Embeddings learn statistical patterns from their training data. If that data contains social stereotypes or unequal associations, the embedding space may reproduce them.

Potentially sensitive relationships can involve:

• Gender
• Race or ethnicity
• Age
• Religion
• Nationality
• Disability
• Occupation
• Socioeconomic status

For example, an embedding space might associate particular occupations more strongly with one demographic group because that pattern appears in the corpus.

Bias evaluation and mitigation are therefore important when embeddings influence decisions about people.

Common Misconceptions

Every coordinate represents an understandable feature

Real embedding dimensions are usually distributed and difficult to name individually.

Similarity means synonymy

Words can be close because they share a topic, category, grammatical function, or typical context.

A smaller vector is automatically more efficient

One-hot inputs are generally represented by indices, not stored as full dense vectors. The primary advantage of embeddings is learned structure.

Two-dimensional plots preserve all relationships

Dimensionality reduction necessarily distorts the original geometry.

Embeddings understand words like humans do

Embeddings capture statistical regularities. They do not by themselves provide grounded human understanding.

Pretrained embeddings are universally appropriate

Representations learned from one domain may perform poorly in another and may carry unwanted biases.

Key Takeaway

One-hot vectors uniquely identify vocabulary items but place every distinct pair of words at the same distance. They cannot express that “apple” is more closely related to “orange” than to “king.”

Word embeddings solve this by learning a dense mapping\[ w\longmapsto e_w\in\mathbb{R}^{d}, \]

where geometric relationships reflect patterns found in text. This helps models share statistical strength across related words and generalize beyond exact phrases observed during training.

Traditional embeddings assign one fixed vector to each word, while modern systems often use subword units and contextual representations. Regardless of the specific method, the central idea remains the same: useful language processing begins with representations that encode more than token identity.