Using Word Embeddings for Transfer Learning in NLP

Word embeddings represent tokens as dense numerical vectors whose geometry captures patterns learned from text. Words that occur in similar linguistic contexts often receive similar representations, allowing a model to transfer knowledge from large collections of unlabeled text to a smaller supervised task.

This form of transfer learning can improve generalization, reduce labeled-data requirements, and provide a far more useful input representation than isolated one-hot vectors.

From One-Hot Vectors to Dense Embeddings

Suppose a vocabulary contains \(V\) words. A one-hot representation of word \(w\) is a vector\[ o_w\in\mathbb{R}^{V} \]

containing one 1 and \(V-1\) zeros.

If \(w\) has vocabulary index \(i\), then\[ (o_w)_j = \begin{cases} 1, & j=i,\\ 0, & j\neq i. \end{cases} \]

One-hot vectors identify words, but they contain no information about relationships between them. For any two different words \(w_i\) and \(w_j\),\[ o_{w_i}^{\top}o_{w_j}=0. \]

Therefore, “apple” is just as different from “orange” as it is from “tractor” in one-hot space.

A word embedding instead represents each word with a dense vector:\[ e_w\in\mathbb{R}^{d}, \]

where \(d\) may be much smaller than \(V\). For example,\[ V=50{,}000, \qquad d=300. \]

The vector is dense and learned from data. Its coordinates do not usually correspond to manually defined properties, but words used in related contexts may occupy nearby regions of the embedding space.

The Embedding Matrix

The embeddings for the complete vocabulary can be stored in a matrix\[ E\in\mathbb{R}^{d\times V}. \]

If \(o_w\) is the one-hot representation of word \(w\), its embedding is\[ e_w=Eo_w. \]

Because multiplying by a one-hot vector simply selects one column, this operation is implemented as an efficient table lookup:\[ e_w=E[:,i]. \]

A model therefore does not normally construct a large one-hot vector explicitly. It uses the token index to retrieve the appropriate row or column from the embedding table.

import torch.nn as nn

embedding = nn.Embedding(
    num_embeddings=vocabulary_size,
    embedding_dim=300
)

embedded_tokens = embedding(token_ids)

import torch.nn as nn

embedding = nn.Embedding(
    num_embeddings=vocabulary_size,
    embedding_dim=300
)

embedded_tokens = embedding(token_ids)

If token_ids has shape\[ (B,T), \]

then embedded_tokens has shape\[ (B,T,300). \]

How Embeddings Support Generalization

Consider a named-entity recognition system trained on:

“Sally Johnson is an orange farmer.”

The model should identify “Sally Johnson” as a person. The surrounding phrase “orange farmer” provides evidence that the sentence describes a human occupation.

Now consider:

“Robert Lin is an apple farmer.”

If the representations for “orange” and “apple” are similar, the model can transfer patterns learned from the first sentence to the second.

A more difficult example is:

“Robert Lin is a durian cultivator.”

The labeled dataset may contain few or no examples involving “durian” or “cultivator.” However, embeddings learned from a much larger text collection may encode relationships such as:\[ e_{\text{durian}} \approx e_{\text{orange}} \approx e_{\text{apple}}, \]

and\[ e_{\text{cultivator}} \approx e_{\text{farmer}}. \]

The task model can then recognize that the new phrase has a structure similar to one it has seen before.

Embeddings allow the model to generalize through similarity of learned representations rather than requiring every relevant word combination to appear in the labeled data.

Why Large Unlabeled Corpora Help

High-quality labeled data is often expensive because people must annotate:

• Names
• Locations
• Sentiment
• Grammatical roles
• Document categories
• Relationships between entities

Unlabeled text is much more abundant. A representation model can learn from large text collections by solving objectives that do not require manual labels, such as:

• Predicting a word from its context
• Predicting surrounding words
• Reconstructing masked tokens
• Predicting the next token

These objectives teach the model statistical relationships between words and contexts.

The resulting representations can then be transferred to a task with a smaller labeled dataset.

Transfer-Learning Workflow

A traditional word-embedding workflow contains three main stages.

Learn or obtain pretrained embeddings

Learn an embedding matrix from a large corpus, or obtain an existing pretrained embedding set.

This stage uses broad linguistic data rather than task-specific labels.

Transfer the embeddings to the target model

Use the pretrained vectors as the input embedding layer of a model for:

• Named-entity recognition
• Sentiment classification
• Text classification
• Information extraction
• Sequence labeling

Instead of receiving one-hot vectors, the model receives dense embeddings:\[ x_t=e_{w_t}. \]

Freeze or fine-tune the embeddings

During supervised training, the embedding matrix may either remain fixed or be updated along with the rest of the model.

This decision depends on the size and relevance of the labeled dataset.

Frozen Embeddings

When the embedding matrix is frozen,\[ E\leftarrow E_{\text{pretrained}}, \]

and gradient descent does not update it.

The task model learns only its remaining parameters.

Advantages

• Reduces the number of trainable parameters
• Lowers the risk of overfitting
• Preserves broad semantic structure
• Works well when labeled data is limited
• Reduces training cost

Disadvantages

• Cannot adapt representations to specialized meanings
• May preserve biases or limitations from the source corpus
• May not match the target domain
• Cannot repair poor representations of important task words

Freezing is often a sensible starting point when the target dataset is small.

Fine-Tuned Embeddings

When embeddings are trainable, the supervised objective updates the matrix:\[ E \leftarrow E-\eta\frac{\partial\mathcal{L}}{\partial E}. \]

This allows the representation of each observed token to adapt to the target task.

Advantages

• Adapts embeddings to domain-specific usage
• Can improve performance with sufficient labeled data
• Allows task-relevant distinctions to emerge

Disadvantages

• Increases overfitting risk
• Can distort useful pretrained relationships
• Rare tokens may receive noisy updates
• Requires more memory for optimizer states and gradients

A common compromise is to begin with frozen embeddings and unfreeze them later using a smaller learning rate.

Freeze or Fine-Tune?

The best decision should be based on validation performance rather than dataset size alone.

Using Different Learning Rates

When embeddings are fine-tuned, they often benefit from a smaller learning rate than newly initialized task layers.

For example,\[ \eta_{\text{embedding}} < \eta_{\text{task}}. \]

This permits gradual adaptation without immediately destroying the pretrained geometry.

A staged approach can be used:

1. Train the task-specific layers while embeddings are frozen.
2. Unfreeze the embedding layer.
3. Continue training with a lower embedding learning rate.
4. Stop based on validation performance.

Why Transfer Helps Most with Limited Labeled Data

Suppose task \(A\) is representation learning from a large text corpus, while task \(B\) is named-entity recognition with a smaller labeled dataset.

Transfer is valuable when:

• The source data is much larger than the target data.
• The source and target contain related linguistic structure.
• The learned representation captures information useful for the target.

The pretrained embedding supplies prior knowledge that the target dataset may be too small to discover independently.

As the target dataset becomes larger, the task model becomes increasingly capable of learning useful representations directly. Pretraining may still help optimization and generalization, but its relative advantage can become smaller.

Static Embeddings

Traditional word embeddings assign one vector to each vocabulary entry:\[ w\longrightarrow e_w. \]

The word “bank,” for example, receives the same vector in both sentences:

• “She deposited money at the bank.”
• “They sat on the bank of the river.”

This is a limitation because the meaning depends on context.

Static embeddings are still useful for:

• Lightweight models
• Small datasets
• Fast retrieval
• Resource-constrained deployment
• Simple similarity features
• Educational demonstrations

However, they cannot directly represent different meanings of the same word in different contexts.

Contextual Embeddings

Modern representation models produce a vector that depends on both the token and its surrounding sequence:\[ e_t = f(w_t,w_1,\ldots,w_T). \]

The representation of “bank” can therefore differ between financial and geographical contexts.

Contextual representations offer:

• Meaning that changes with context
• Better handling of ambiguity
• Richer syntactic information
• Strong transfer across many downstream tasks

Instead of transferring only a lookup table, modern transfer learning often transfers an entire pretrained encoder.

The supervised model may then:

• Freeze the encoder and train a small prediction head
• Fine-tune the complete encoder
• Update only selected adapter or low-rank parameters

Subword Representations

A fixed word vocabulary creates difficulty for rare or unseen words. Subword tokenization addresses this by dividing text into reusable components.

An uncommon term can be represented by several known subword units rather than a single <UNK> token.

This has several benefits:

• Better handling of unseen words
• Smaller vocabulary requirements
• Shared representations across related word forms
• Improved support for productive morphology
• More robust treatment of names and technical terms

For named-entity recognition, subword units are especially valuable because names frequently fall outside a conventional word vocabulary.

Applying Embeddings to Sequence Labeling

Suppose the input sentence contains tokens\[ w_1,w_2,\ldots,w_T. \]

First, retrieve their embeddings:\[ e_t=E[w_t]. \]

These vectors become the inputs to a sequence encoder:\[ h_t=f(h_{t-1},e_t). \]

For a bidirectional model,\[ \overrightarrow{h}_t = f_{\rightarrow} (\overrightarrow{h}_{t-1},e_t), \]\[ \overleftarrow{h}_t = f_{\leftarrow} (\overleftarrow{h}_{t+1},e_t). \]

The contextual state is\[ h_t = [ \overrightarrow{h}_t; \overleftarrow{h}_t ]. \]

A classifier then predicts a label for every token:\[ \hat{y}_t = \operatorname{softmax}(W_yh_t+b_y). \]

Bidirectionality is useful because identifying a token often requires information from both sides.

Word Embeddings and Face Embeddings

Word embeddings share a conceptual relationship with embeddings used in image and face systems. Both map complex or discrete objects into dense vector spaces where geometric relationships become meaningful.

However, traditional implementations differ.

Traditional word embeddings

A finite vocabulary is associated with a table:\[ w_i\longrightarrow e_i. \]

The representation is retrieved using the token’s index.

Face embeddings

A neural encoder computes a representation from image pixels:\[ x\longrightarrow f(x). \]

It can generate an embedding for a new face image that was never part of the training set.

Modern contextual token encoders are more similar to image encoders because they compute representations dynamically rather than retrieving a single fixed vector.

Measuring Embedding Similarity

Cosine similarity is commonly used to compare embeddings:\[ \operatorname{sim}(u,v) = \frac{u^\top v} {\lVert u\rVert_2\lVert v\rVert_2}. \]

A value close to 1 indicates similar directions, while a value near 0 indicates weak directional similarity.

However, nearest neighbors in an embedding space should be interpreted carefully:

• Similarity may reflect context rather than strict synonymy.
• Antonyms may appear close because they occur in similar sentences.
• Frequency effects can distort geometry.
• Social biases in the corpus can be encoded in the vectors.
• Geometric similarity does not guarantee usefulness for every target task.

Domain Mismatch

Pretrained embeddings are most useful when their source corpus resembles the target domain.

For example, general web embeddings may represent everyday language well but handle specialized medical terminology poorly.

Possible responses include:

• Continue representation learning on unlabeled domain text
• Fine-tune embeddings using labeled target data
• Expand the vocabulary
• Use subword representations
• Train domain-specific embeddings
• Combine general and domain-specific representations

The choice depends on the amount of available data and the severity of the domain mismatch.

Common Mistakes

Assuming nearby vectors are always synonyms

Words with related contexts may be close even when their meanings differ or oppose one another.

Fine-tuning aggressively on a small dataset

Large updates can destroy useful pretrained structure and lead to overfitting.

Ignoring missing vocabulary items

A model needs a clear strategy for unseen tokens, such as subword decomposition or an unknown-token representation.

Treating embeddings as neutral

Embeddings can preserve stereotypes and other biases present in their training corpus.

Comparing one-hot and dense vectors only by dimension

One-hot vectors are sparse identifiers. Dense embeddings contain learned relational information. Their value is not merely that they are shorter.

Ignoring sequence context

A static embedding alone cannot determine the intended meaning of an ambiguous word. A contextual sequence model is still needed.

Key Takeaway

Word embeddings replace sparse token identifiers with dense learned representations:\[ e_w=Eo_w. \]

Because embeddings are learned from large text collections, related words can receive related vectors. A supervised NLP model can transfer this structure to a smaller labeled dataset, allowing patterns learned for familiar expressions to generalize to rarer ones.

Traditional static embeddings associate one fixed vector with each vocabulary item. Modern systems commonly extend this idea with subword tokenization and contextual encoders that compute different representations according to surrounding text.

When labeled data is limited, freezing pretrained representations can reduce overfitting. With more target data or substantial domain mismatch, careful fine-tuning can adapt those representations to the task.